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ESTABLISHING  AN  INTELLECTUAL  AND  THEORETICAL  FOUNDATION  FOR 
THE  AFTER  ACTION  REVIEW  PROCESS  -  A  LITERATURE  REVIEW 


EXECUTIVE  SUMMARY 


Research  Requirement 

The  purpose  of  this  report  is  to  provide  a  literature  review  of  the  cognitive  and 
learning  science  research  that  is  relevant  to  defining  an  effective  after  action  review 
(AAR)  process.  The  goal  of  this  research  review  is  to  assemble  research  sources  that 
apply  to  the  design  and  conduct  of  AARs.  Therefore,  this  report  provides  a  synopsis  of 
research  that  exists,  identifies  notable  researchers  who  have  addressed  the  problem, 
presents  results  of  both  military  and  non-military  investigations  or  implementations  of  the 
AAR  technique,  highlights  existing  theories  that  may  contribute  to  the  advancement  of 
AARs,  and  isolates  specific  areas  that  demand  further  work. 

Procedure 

AARs  are  meant  to  serve  a  pedagogical  purpose:  to  enhance  learning  from 
experience.  As  such,  their  structure  and  function  is  reminiscent  of  a  number  of 
theoretical  domains.  Such  domains  may  include: 

•  Mainstream  Psychology  (particularly  with  regard  to  the  optimization  of  feedback, 
training  methodology,  and  the  behavior  of  small  groups  and  teams) 

•  Education  (especially  curriculum  development  and  the  behavior  of  facilitators) 

•  Instructional  Technology  (as  concerns  the  development  and  incorporation  of 
methods  by  which  performance  data  may  be  gathered  and  replayed  to 
participants) 

•  Industrial  Psychology  (for  the  design  of  performance  appraisal  systems,  the  study 
of  leadership,  and  the  role  of  organizational  influences) 

•  Knowledge  Management  (an  interdisciplinary  endeavor  that  reflects  organization 
of  curricular  issues  and  their  sequential  presentation  and  mastery) 

•  Military  Planning 

•  Leadership. 

Previous  attempts  to  understand  the  AAR  process  have  typically  restricted 
themselves  to  a  subset  of  these  areas,  or  have  taken  a  decidedly  theoretical  position. 
However,  consideration  of  theory  is  necessary  so  as  to  avoid  reinventing  the  wheel  and  to 
ensure  that  the  AAR  represents  a  flexible  and  powerful  method  for  optimizing 
operational  learning. 
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Because  of  the  diversity  of  theoretical  contributors  to  AARs,  it  has  been  difficult 
to  fonnulate  one  master  list  of  best  practices.  Rather,  what  is  needed  is  a  theoretical 
model  that  reflects  the  myriad  of  discipline  influences,  and  that  is  flexible  enough  to 
apply  to  diverse  operational  circumstances.  To  date,  few  theoretical  models  have  been 
proposed.  Those  that  have  been  proposed  suffer  from  lack  of  validation.  Thus  the  irony: 
even  though  AARs  are  meant  to  be  the  pinnacle  of  a  “learning  organization,”  there  have 
been  few  documented  attempts  to  specify  the  principles  of  learning  or  knowledge 
management  to  the  conduct  of  AAR  sessions. 

Findings 

The  report  by  Newlin  and  Bliss  (2007)  included  an  expansive  description  of  a 
number  of  research-driven  recommendations  for  the  optimal  conduct  of  AARs.  Further 
conclusions  are  drawn  from  the  research  that  is  presented  here.  The  following  bullet 
points  summarize  the  significant  theoretical  information  that  emerged  from  this  work. 

•  AAR  designers  and  facilitators  should  arrange  and  implement  AAR  sessions  to 
ensure  the  prompt  encoding,  robust  storage,  and  ready  retrieval  of  task  relevant 
information  by  trainees.  An  example  of  such  a  process  may  involve  exploiting 
human  tendencies  for  spreading  activation  by  pointing  out  relationships  among 
mission  items,  goals  and  procedures. 

•  To  enable  deep  and  lasting  processing  of  mission  information,  AAR  facilitators 
should  arrange  for  presentation  of  “ground  truth”  from  a  variety  of  perspectives. 

•  Facilitators  should  use  hierarchical  review  of  mission  events  to  cater  to  the 
development  of  cognitive  scripts,  mental  models,  and  cognitive  maps. 

•  To  prevent  false  memory  formation,  facilitators  should  encourage  checks  and 
balances  among  team  members,  along  with  presentations  of  recorded  mission 
action. 

•  Facilitators  should  take  care  to  address  the  need  for  shared  situation  awareness 
among  team  members. 

•  Team  members  should  be  encouraged  to  express  the  benefits  gained  by  review  of 
team  perfonnance. 

•  Demands  for  divided  attention  should  be  balanced  with  the  growth  of  expertise  by 
presenting  material  in  a  graduated  (adaptive)  and  distributed  fashion. 

•  Facilitators  should  be  flexible  in  team  assignments  and  feedback,  focusing  on 
dyads  or  smaller  team  units  to  illustrate  important  points. 

•  Facilitators  and  team  members  should  be  sensitive  to  communication 
requirements  and  breakdowns.  Social  dynamics  such  as  attributions,  conflict,  and 
cooperation  are  central  to  a  team’s  ultimate  success. 
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•  Feedback  sessions  should  allow  for  self-feedback  and  task  feedback  possibilities. 

•  Facilitators  should  be  aware  of  the  potential  for  negative  reactions  to  feedback, 

and  plan  accordingly. 

•  Facilitators  should  ensure  that  all  team  members  accept  the  feedback  given  during 

the  AAR  session.  If  dissent  exists,  work  to  resolve  the  conflict. 

With  the  complexity  of  the  AAR  process  and  the  advancements  being  made  in 
simulation  and  training  and  automated  task  delegation,  we  believe  that  additional 
research  is  warranted  in  several  areas.  In  some  of  these  areas,  research  has  been 
conducted  by  a  limited  number  of  scientists,  trainers,  and  institutions.  In  others,  little 
work  has  been  attempted: 

Distributed  AAR.  Modern  simulation  technology  has  offered  the  capability  to 
train  tasks  to  teams  that  are  geographically  distributed.  Yet,  AARs  are  a  piece  of  the 
training  puzzle  that  has  yet  to  be  well  refined  for  application  to  geographically  dispersed 
teams. 

Shared  Mental  Models.  As  we  investigated  the  relevant  literature  with  regard  to 
cognition,  team  training  and  situation  awareness,  we  realized  that  these  three  areas  were 
of  extreme  importance  to  the  success  of  AARs,  and  that  they  intersected  though  one 
major  construct:  shared  mental  model. 

Shared  Situation  Awareness.  It  is  clear  that  a  desired  product  of  the  AAR  session 
is  a  common  appreciation  for  the  battle  situation  and  the  influences  on  it.  Of  particular 
importance  are  the  various  levels  of  situation  awareness  and  how  they  may  change  given 
tactics  and  strategies  employed  by  the  trainees.  Investigating  shared  situation  awareness 
may  entail  examinations  of  technology-driven  environmental  displays  (of  terrain,  forces, 
or  supplies)  and  may  require  integrated  consideration  of  past  performance  and  events  as 
they  relate  to  current  performance. 

Leadership.  Within  this  report,  we  have  made  several  recommendations  that 
concern  the  leadership  behaviors  demonstrated  by  AAR  facilitators.  The  AAR  process 
bears  a  similarity  to  a  number  of  leadership  situations  such  as  sports  teams  or  business- 
related  “tiger  teams”,  where  problems  are  relatively  well-defined  and  members  have 
acknowledged  areas  of  expertise.  Future  researchers  may  do  well  to  explore  the  practice 
of  effective  leaders  so  that  facilitator  actions  may  be  prescribed  and  perhaps  ultimately 
automated. 

Effectiveness  of  AAR.  Though  the  AAR  has  been  an  Army  tradition  for  more 
than  30  years,  investigations  of  its  effectiveness  have  been  few  and  anecdotal.  Such 
investigations  must  go  beyond  collecting  subjective  opinions  and  general  statements  of 
approval  voiced  by  command  elements.  Rather,  researchers  must  complete  a  program  of 
transfer  of  training  studies  to  show  that  variations  in  AAR  design  and  conduct  will  cause 
differences  in  individual  and  collective  task  performance. 
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such,  they  would  be  of  interest  to  researchers  who  wish  to  acquire  or  update  their 
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ESTABLISHING  AN  INTELLECTUAL  AND  THEORETICAL  FOUNDATION  FOR 
THE  AFTER  ACTION  REVIEW  PROCESS  -  A  LITERATURE  REVIEW 


Overview 

Modern  warfare  is  increasingly  complex.  Historically,  battles  could  be 
predictably  counted  on  to  occur  on  a  large  battlefield  with  readily  identifiable  opponents, 
weapons,  tactics  and  strategies.  However,  in  recent  years,  the  scope  of  warfare  has 
changed.  Today’s  Soldiers  must  prepare  to  fight  using  unconventional  techniques  against 
unpredictable  and  adaptable  enemies.  Furthermore,  the  time  available  to  train  is  often 
constricted,  so  that  Soldiers  must  adopt  techniques  quickly,  and  must  leam  to  work 
together  as  a  team  (Kerrick,  2005). 

Teamwork,  especially,  is  a  crucial  element  of  battle  planning  and  execution. 
Military  peacekeeping  and  peace  enforcement  campaigns  in  theatres  such  as  Bosnia, 

Haiti,  Iraq,  and  Afghanistan  have  required  Soldiers  from  all  branches  of  the  U.S.  military 
to  come  together  and  cooperate.  Such  a  joint  philosophy  is  to  many  an  unfamiliar 
concept,  particularly  because  Soldiers  are  generally  trained  at  centers  that  reflect  their 
branch  of  service  (Army,  Navy,  Air  Force,  Marines).  It  is  imperative  that  military 
training  centers  adopt  training  methods  that  enhance  trainees’  willingness  and  ability  to 
adopt  complex  skills  quickly  and  that  foster  their  capacity  for  teamwork  on  the  battlefield 
(Lowrey,  1999). 

Increasingly,  the  military  has  turned  to  simulation  as  a  way  to  represent  skill  sets 
that  are  complex,  dangerous,  or  costly  to  train.  Historically,  it  has  been  common  to  teach 
teamwork  and  battle  skill  execution  using  physical  mockups  and  devoted  environments 
for  practice  (such  as  the  National  Training  Center).  In  recent  years,  the  concept  of  battle 
task  simulation  has  evolved  so  that  trainees  interact  not  just  with  physical  simulators  (as 
in  the  case  of  specific  weapons  or  transportation  systems)  but  with  local  area  and 
distributed  virtual  environments  (especially  for  dismounted  infantry).  Regardless  of  the 
form  of  simulation  employed,  however,  a  central  challenge  has  been  the  capture  of  action 
during  a  training  exercise  so  that  trainees  may  review  it  and  be  taught  to  improve 
performance. 

Researchers  have  for  many  years  acknowledged  provision  of  feedback  as  essential 
for  efficient  learning  (Mory,  1992).  Yet,  the  methods  available  for  administering 
feedback  vary  widely,  as  does  the  consistency  with  which  human  trainers  and  automated 
training  systems  accomplish  feedback  administration.  Unfortunately,  failures  to 
consistently  and  faithfully  administer  meaningful  training  feedback  have  led  to  poor 
knowledge  retention,  poor  knowledge  transfer,  and  a  tendency  among  some  to  discount 
the  importance  of  the  feedback  process.  Some  of  these  failures  are  undeniably  caused  by 
difficulties  capturing  meaningful  data.  However,  more  often,  poor  feedback  stems  from 
the  intuitive  design  of  feedback  sessions  and  variability  in  feedback  administration. 
Consequently,  trainees  who  could  benefit  from  proper  feedback  administration  may  adopt 
a  cavalier  or  skeptical  opinion  of  the  importance  of  training  feedback.  The  complexity  of 
modem  military  missions  and  the  necessity  for  seamless  interaction  of  Soldier  team 
members  requires  that  feedback  be  designed  consistently  and  effectively,  and  that  it  be 
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delivered  to  ensure  maximal  retention  and  task  transfer  (Baird,  Holland,  &  Deacon, 
1999). 


After  Action  Review  Definition 

Within  and  outside  of  military  circles,  the  provision  of  training  feedback  has  been 
termed  “after  action  review  (AAR).”  Even  though  the  notion  of  review  of  training 
success  has  arguably  been  considered  since  Alexander  the  Great  standardized  military 
training,  the  modem  definition  of  this  tenn  originates  with  the  U.S.  Army’s 
conceptualization  of  the  AAR  technique  in  the  1970s.  Following  the  Viet  Nam  war,  the 
Army  reflected  upon  its  status;  morale  among  troops  was  low,  and  battle  missions  had 
not  progressed  as  intended  (Darling,  Parry,  &  Moore,  2005).  Development  of  the  AAR 
concept  was  intended  to  enable  quality  improvement  consistent  with  the  restructuring  of 
the  Army  during  the  1980s  and  1990s.  Subsequently,  the  AAR  technique  has  been 
incorporated  into  training  practice  and  is  now  a  formal  element  of  all  training 
(Department  of  the  Army,  1993).  Most  recently  the  Army  published  FM  6-01.1,  titled 
Knowledge  Management  Section,  which  stresses  the  importance  of  the  conduct  of  the 
AAR  as  an  integral  part  of  operations.  Appendix  B  of  FM  6-01.1  is  devoted  to  a  detailed 
discussion  of  the  process  for  preparing  and  conducting  AAR  for  use  as  a  learning 
technique  during  operations  (Department  of  the  Army,  2008). 

The  AAR  is  a  reflective  session  that  is  designed  to  allow  trainees  to  review  prior 
training  performance.  The  philosophy  of  AARs  is  pedagogical.  The  sessions  are  not 
intended  to  be  critical  evaluations  of  performance  so  much  as  collaborative  meetings  held 
to  enhance  learning.  Ideally,  AARs  should  be  conducted  to  focus  on  what  was  planned  to 
happen  during  training,  what  actually  did  happen  during  training,  why  events  unfolded  as 
they  did,  and  what  might  be  modified  during  subsequent  training  or  perfonnance 
sessions.  It  is  important  to  emphasize  that  AARs  are  generally  conducted  at  the  team 
level.  Though  individual  AARs  are  possible,  they  are  typically  introspective  and 
personal.  Optimally,  AAR  sessions  should  be  held  as  soon  after  training  as  is  practical. 

In  practice,  AARs  vary  in  tenns  of  their  fonnality.  Formal  sessions  may  be 
supplemented  by  individual  debriefings,  collective  “hot  washes,”  and  periodic  reviews  of 
success  toward  objectives.  An  important  element  of  AARs  is  the  role  of  the  facilitator. 
This  person  (ideally  a  training  expert  who  is  also  familiar  with  the  goals  of  the  training 
session)  leads  the  trainees  through  the  session,  guiding  discussion  and  ensuring  that  the 
pedagogical  and  collaborative  spirit  of  the  AAR  is  not  forsaken  and  that  participants 
accrue  the  knowledge  that  is  intended. 

The  above  discussion  highlights  the  intended  practice  and  implementation  of 
AARs.  However,  there  is  considerable  variability  with  regard  to  how  AARs  are 
conducted  in  the  field.  Much  of  this  variability  reflects  the  complexity  of  the  training 
situation.  Many  training  sessions  (particularly  at  formal,  full-fledged  facilities  such  as 
the  National  Training  Center)  may  stretch  for  two  weeks  or  more,  as  Soldiers  engage  in 
simulated  combat  against  thinking,  flexible,  creative  opponents.  Over  that  period,  there 
are  likely  to  be  many  iterations  of  AAR,  from  less  formal  “hot  washes”  to  more  formal 
collaborative  sessions. 
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An  important  element  that  contributes  to  the  variability  of  the  AAR 
implementation  is  the  AAR  facilitators  (Keene,  1994).  Each  one  may  have  a  different 
philosophy  about  how  best  to  approach  the  team,  and  may  have  a  different  interaction 
style.  Similarly,  different  facilitators  may  be  more  or  less  effective  at  stimulating 
discussion,  leading  participants,  structuring  the  lessons,  reflecting  on  planned  goals,  and 
synthesizing  the  lessons  learned  from  the  AAR.  Trainees,  themselves,  also  constitute  a 
source  of  variability.  Some  groups  function  as  a  cohesive  unit  with  a  clear  leader.  Other 
groups  may  be  relatively  unfamiliar  with  each  other,  may  have  a  less  effective  leader,  or 
may  suffer  from  internal  strife  or  conflict.  Ultimately,  the  AAR  process  resembles  a  sort 
of  dance,  where  the  trainees  and  facilitator  must  share  responsibility  for  teasing  out 
lessons  learned  from  the  training  exercise.  The  music  they  dance  to  may  represent  the 
tempo  of  battle,  the  coordination  with  other  allied  teams,  the  awareness  of  overall  goals, 
and  the  progress  made  toward  the  overall  objectives. 

Each  of  these  issues  has  rendered  the  AAR  a  highly  important  but  inconsistent 
and  potentially  unpredictable  tool.  The  following  quote  from  Peter  Senge  (2001) 
illustrates  the  first  basic  problem: 

The  Army's  After  Action  Review  (AAR)  is  arguably  one  of  the  most  successful 
organizational  learning  methods  yet  devised.  Yet,  most  every  corporate  effort  to 
graft  this  truly  innovative  practice  into  their  culture  has  failed  because,  again  and 
again,  people  reduce  the  living  practice  of  AAR's  to  a  sterile  technique. 

This  suggests  that  the  AAR  technique  is  more  than  simply  a  process  that  is 
transferable  across  organizations  and  contexts.  Rather  it  should  be  tailorable  for  the 
needs  of  particular  groups  who  adopt  it. 

A  second  problem  with  current  AAR  practices  is  that  they  are  frequently  defined 
by  practical  constraints,  rather  than  findings  from  behavioral  researchers.  AARs  have 
been  referred  to  as  a  self-contained  substantiation  of  a  broader  knowledge  management 
program.  As  such,  they  are  designed  with  the  overriding  purpose  of  increasing 
knowledge  gained.  However,  to  realize  the  benefits  that  are  possible  from  AARs,  it  is 
necessary  to  learn  from  researchers  who  have  devoted  their  efforts  to  optimizing  learning. 
Such  researchers  have  published  volumes  devoted  to  curriculum  design,  practice, 
feedback  and  other  theoretical  concerns.  Yet,  too  often  the  implementation  of  AARs 
caters  to  time  constraints,  leadership  constraints,  labor  constraints,  and  monetary 
constraints.  As  a  result,  the  value  of  the  technique  is  distilled  or  eliminated  altogether. 

As  an  illustration  of  these  problems,  DeGrosky  (2005)  highlighted  several  problems 
faced  by  Wildland  Fire  Agencies  as  they  worked  to  incorporate  the  AAR  process. 

Among  the  specific  problems  DeGrosky  noted  were  following  AAR  technique  without 
context,  irregular  use  of  AARs,  practicing  infonnal  or  unstructured  AARs,  engaging  in 
unsystematic  preparation  for  AARs,  and  not  properly  training  AAR  facilitators. 

A  third  problem  with  the  conduct  of  AARs  reflects  the  theoretical  grounding  of 
the  technique.  AARs  are  meant  to  serve  a  pedagogical  purpose:  to  enhance  learning 
from  experience.  As  such,  their  structure  and  function  is  likely  reminiscent  of  a  number 
of  theoretical  domains.  Such  domains  may  include  mainstream  psychology  (particularly 
with  regard  to  the  optimization  of  feedback,  training  methodology,  and  the  behavior  of 
small  groups  and  teams),  education  (especially  curriculum  development  and  the  behavior 
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of  facilitators),  instructional  technology  (as  concerns  the  development  and  incorporation 
of  methods  by  which  perfonnance  data  may  be  gathered  and  replayed  to  participants), 
industrial  psychology  (for  the  design  of  performance  appraisal  systems,  the  study  of 
leadership,  and  the  role  of  organizational  influences),  knowledge  management  (an 
interdisciplinary  endeavor  that  reflects  organization  of  curricular  issues  and  their 
sequential  presentation  and  mastery)  and  military  planning  and  leadership.  Previous 
attempts  to  understand  the  AAR  process  have  typically  restricted  themselves  to  a  subset 
of  these  areas,  or  have  taken  a  decidedly  a  theoretical  position.  However,  consideration 
of  theory  is  necessary  so  as  to  avoid  reinventing  the  wheel  and  to  ensure  that  the  AAR 
represents  a  flexible  and  powerful  method  for  optimizing  operational  learning. 

Because  of  the  diversity  of  theoretical  contributors  to  AARs,  it  has  been  difficult 
to  fonnulate  one  master  list  of  best  practices.  Rather,  what  is  needed  is  a  theoretical 
model  that  reflects  the  myriad  of  discipline  influences,  and  that  is  flexible  enough  to 
apply  to  diverse  operational  circumstances.  To  date,  few  theoretical  models  have  been 
proposed.  Those  that  have  been  proposed  suffer  from  lack  of  validation.  Thus  the  irony: 
Even  though  AARs  are  meant  to  be  the  pinnacle  of  a  “learning  organization,”  there  have 
been  few  documented  attempts  to  specify  the  principles  of  learning  or  knowledge 
management  to  the  conduct  of  AAR  sessions. 

The  goal  of  this  research  review  is  to  assemble  research  sources  that  apply  to  the 
design  and  conduct  of  AARs.  Our  research  team  is  concerned  mostly  with  academic 
research  that  pertains  to  the  topics  mentioned  earlier.  Therefore,  the  following  sections 
will  provide  a  synopsis  of  research  that  exists.  We  also  plan  to  review  that  research  in  an 
attempt  to  identify  notable  researchers  who  have  addressed  the  problem,  present  results 
of  both  military  and  non-military  investigations  or  implementations  of  the  AAR 
technique,  highlight  existing  theories  that  may  contribute  to  the  advancement  of  AARs, 
and  isolate  specific  areas  that  demand  further  work. 


Research  Approach  Followed 

The  approach  that  the  research  team  followed  demonstrates  several  priorities. 
First,  it  was  necessary  for  the  researchers  to  understand  the  AAR  process  as  it  was 
initially  envisioned  by  the  U.S.  Army,  as  it  has  evolved  since  its  inception,  as  it  is 
currently  practiced,  and  as  it  must  change  to  fit  foreseeable  demands.  Second,  it  is 
important  for  the  literature  review  to  comprehensively  focus  on  a  relevant  variety  of 
academic  disciplines.  Therefore,  we  emphasized  available  academic  research  from 
conventional  article  and  report  databases.  We  also  stipulated  a  number  of  search  tenns 
that  corresponded  to  the  functions,  qualities  and  challenges  of  the  AAR  process. 

Because  our  research  team  had  developed  a  theoretical  model  of  AAR  (see 
Mastaglio,  Jones,  Bliss,  &  Newlin,  2007),  we  considered  that  model  as  a  basic  starting 
point  for  our  investigation.  The  Integrated  Theory  of  After  Action  Review  (ITAAR) 
model  (see  Figure  1)  stipulated  several  theoretical  areas  that  are  presumed  central  to 
AAR  development  and  implementation.  The  current  literature  review  in  this  report  was 
intended  to  supplement  the  research  that  led  to  the  development  of  ITAAR.  Toward  that 
end,  part  of  the  current  report  will  summarize  past  research  (leading  to  ITAAR)  and  will 
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present  additional  evidence  in  these  areas.  Furthermore,  we  will  discuss  literature  that 
reflects  application  of  AARs  in  military  and  industrial  environments. 


Reliance  on  Prior  Research 

As  we  assembled  relevant  research  articles,  we  paid  particular  attention  to 
individuals  and  research  centers  that  have  been  notably  influential  or  active  in  the  study 
of  AAR.  To  do  so,  we  searched  research  databases  to  determine  those  articles  and 
authors  that  were  commonly  cited  by  others.  Our  hope  was  that  reviewing  these  sources 
would  allow  us  to  determine  whether  the  most  prolific  authors  in  the  field  had  attempted 
to  explore  theoretical  foundations  of  the  AAR  process. 

Our  investigative  process  focused  mostly  on  established  research  databases, 
including  PsychINFO,  ERIC,  Google  Scholar,  the  Defense  Technical  Information  Center, 
and  Applied  Science  and  Technology.  After  we  initially  focused  on  research  specifically 
targeting  AAR,  we  then  explored  four  key  areas  (inspired  by  IT  AAR):  learning  theory, 
team  dynamics,  feedback,  and  relevant  cognitive  processes  (examples  within  the  latter 
category  include  reward,  reinforcement,  and  memory).  We  employed  a  number  of  search 
tenns  to  ensure  that  our  investigation  was  powerful  enough  to  uncover  existing  research. 
Examples  of  these  search  terms  include  “feedback,”  “after  action  review,”  “performance 
review,”  “learning  from  experience,”  “knowledge  management,”  and  “action  learning.” 

Our  search  led  us  to  over  150  sources  that  we  considered  central  to  the  design  and 
implementation  of  AAR.  They  are  listed  in  the  bibliography  section  at  the  end  of  this 
report.  After  identifying  these  sources,  we  then  obtained  them  and  created  an  online 
database  to  organize  those  considered  most  directly  influential.  We  posted  summaries  of 
the  obtained  articles  using  a  publicly  available  spreadsheet  tool  sponsored  by  Google 
(http://spreadsheets. google. com/pub?key=pNt_AV5mAcxT41ABZHDSpSQ).  Within 
that  spreadsheet,  we  provided  a  summary  of  each  article  as  well  as  key  identifiers  to 
assist  us  in  the  classification  of  each  source.  Concurrently,  we  constructed  a  synopsis  of 
research  in  each  domain  area  in  the  form  of  bullet  points  to  drive  our  subsequent 
discussion  (http://docs. google. com/Doc?id=dgw9kpjw_22d3dmsg4w).  Ultimately,  we 
categorized  and  organized  our  sources  according  to  several  criteria:  the  application  area 
(military,  academic,  industrial),  theoretical  domain  (learning  theory,  team  dynamics, 
feedback,  or  cognitive  processes),  and  application  field  (specific  application  area). 
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-  Knowledge  of  task  facilitates  goal  setting;  goals  structure  the  knowledge 
dispersement  process.(9) 

-  Feedback  is  formalized  within  performance  appraisals;  performance  appraisals 
are  the  mechanism  for  feedback. (10) 

1  Wiener  (1948) 

2  Saavedra,  Earley  &  Van  Dyne  (1993);  Matsui,  Kakuyama  &  Onglatco  (1987) 

3  Mager  (1984) 

4  Cannon-Bowers  &  Salas  (2001) 

5  Smith  (2001) 

6  Schmidt  &  Kle inbeck  (1990) 

7  Jones  (1997) 

8  Dyer  (1986) 

9  Dyer  (1986);  Saavedra  etal.  (1993) 

10  Patrick  (1992) 


Figure  1.  Integrated  theory  of  after  action  review  (Newlin  &  Bliss,  2007). 
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Detailed  Results  Organized  by  Source,  Domain,  and  Field 


In  our  examination  of  the  pertinent  literature  available,  we  detennined  that  there 
were  several  researchers  and  research  centers  that  appeared  to  be  particularly  active  with 
regard  to  investigations  of  AAR.  Some  of  these  centers  and  individuals  have  focused  on 
the  history  of  AARs,  some  have  concentrated  on  the  structure  and  implementation  of 
AARs,  some  have  developed  or  modified  the  underlying  theories  that  support  AAR 
design,  and  some  have  focused  on  the  variability  of  AARs  given  particular  goals.  To  be 
relevant  for  our  review,  each  source  was  required  to  have  focused  on  ultimate 
optimization  of  AARs.  Our  research  team  valued  this  above  other  criteria  because  of  the 
noted  failures  of  the  AAR  process  to  achieve  goals  of  information  transfer  and  learning 
from  experience  (DeGrosky,  2005). 

The  agencies  or  organizations  that  have  concerned  themselves  with  AARs  span 
government  as  well  as  industry.  Each  of  the  armed  forces  has  adopted  their  own  version 
of  AAR,  with  the  U.S.  Army  leading  the  way.  Perhaps  the  most  prolific  military 
organizations  to  study  AAR  have  been  the  U.S.  Army  Research  Institute  and  the  Center 
for  Army  Lessons  Learned  (CALL).  These  groups  have  produced  many  reports  and 
guides  that  describe  methods  for  conducting  AARs  and  recommendations  for  their 
effective  implementation  and  facilitation.  The  Army’s  Functional  Area  (FA)  57, 
Simulation  Operations,  proponency  office  published  an  extensive  DVD-based  AAR 
Toolkit  (2003)  that  provides  multi-media  instruction,  recommendations  and  example 
templates  for  trainers  to  use  in  the  preparation  and  facilitation  of  AARs  (FA-57 
Proponency,  2003).  This  toolkit  was  not  widely  distributed  beyond  FA  57  personnel  so  it 
had  relatively  little  impact  on  the  large  population  of  Army  trainers. 

As  we  searched  for  relevant  research  bearing  on  the  success  of  the  AAR 
procedure,  our  research  team  concerned  itself  first  with  the  identification  of  critical  or 
influential  articles  discussing  AARs.  Perhaps  the  most  widely  cited  resource  is  Morrison 
and  Meliza’s  (1999)  foundations  report.  That  report  discusses  the  history,  structure  and 
function  of  AARs.  It  also  discusses  the  theoretical  grounding  for  AAR  and  the  need  for 
research  in  particular  areas.  Darling,  Parry  and  Moore’s  (2005)  article  in  the  Harvard 
Business  Review  examines  the  practice  of  AAR,  discussing  the  implications  and  potential 
benefits  of  extrapolating  the  AAR  technique  to  the  business  world.  There  have  also  been 
academic  works  such  as  Justin  Gubler’s  (1997)  doctoral  dissertation  at  the  University  of 
Central  Florida.  That  effort  addressed  the  AAR’s  role  within  unit  simulation  training. 
Also,  we  consider  Salter  and  Klein’s  (2007)  research  report  to  be  highly  relevant  and 
timely  for  our  effort.  Finally,  we  believe  that  the  recent  work  by  the  Wildland  Fire  Safety 
Association  (see  DeGrosky,  2005)  reveals  important  truths  about  the  applicability  of  the 
AAR  procedure  to  non-military  domains. 

Our  search  for  information  relevant  to  the  design  and  implementation  of  AARs 
led  us  to  consider  several  supporting  theoretical  areas  within  the  psychological, 
educational,  and  social  research  fields.  From  our  review  of  the  AAR  research  that  has 
been  conducted  in  the  past,  we  considered  the  most  relevant  academic  literature  areas  to 
be  perfonnance  appraisal  design,  development  of  instructional  technology,  human 
learning,  task  training,  teamed  or  group  performance,  leadership,  design  and 
implementation  of  feedback  (knowledge  of  results  or  performance),  and  knowledge 
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management.  In  subsequent  sections  of  this  report,  we  will  present  summarized  findings 
from  our  prior  work  in  many  of  these  areas.  This  report  adds  to  that  previous  literature 
review  by  presenting  additional  relevant  literature  in  the  areas  of  human  cognition,  team 
dynamics,  and  feedback. 

In  each  of  the  areas  mentioned  above,  there  appeared  to  be  seminal  articles  that 
drove  our  investigation  forward.  Those  sources  are  listed  below  within  Table  1.  It  is 
important  to  emphasize  that  this  is  not  an  exhaustive  list,  as  each  area  has  within  it  a 
plethora  of  sources  available. 


Summary  of  Findings 

As  mentioned  above,  our  findings  in  this  report  are  meant  to  supplement  research 
conducted  earlier  in  support  of  the  development  of  the  Integrated  Theory  of  After  Action 
Review  (ITAAR,  Mastaglio  et  ah,  2007). 

In  their  report,  Newlin  and  Bliss  (2007)  stipulated  several  recommendations  for 
the  AAR  process,  driven  by  research  in  each  of  the  relevant  research  fields  (see  Figure  1). 
We  include  those  recommendations  in  toto  as  Appendix  A  of  this  report,  and  consider 
them  as  a  starting  point  for  the  current  literature  review.  The  research  that  was  completed 
by  Newlin  and  Bliss  (2007)  led  to  a  number  of  conclusions  and  recommendations 
pertinent  to  the  improvement  of  the  AAR  process.  These  are  listed  below.  First  are 
presented  recommendations  concerning  the  facilitator’s  role,  followed  by 
recommendations  relevant  to  the  trainees  and  the  AAR  process  in  general. 

•  Based  on  several  integrated  theories  about  a  leader’s  role  in  providing  feedback, 
facilitators  should  be  experienced  leaders  who  cater  to  learners  by  providing 
knowledge  of  results. 

•  Facilitators  should  implement  an  instructional  plan  that  includes  challenging,  yet 
attainable  goals.  These  goals  should  be  specified  before  training  occurs. 

•  Facilitators  who  feel  capable  in  their  communication  skills  and  knowledge  evoke 
better  perfonnance  from  trainees.  This  efficacy  should  be  evident  within  the 
AAR  process.  Facilitator  training  should  focus  on  building  leadership  efficacy, 
communication  skills  and  interpersonal  confidence. 

•  Facilitators  in  training  should  assist  with  the  development  and  implementation  of 
AARs  before  leading  an  AAR  process  alone.  Such  a  practice  would  build  their 
knowledge  of  the  process  and  their  confidence. 

•  Facilitators  who  possess  implicit  knowledge  must  be  encouraged  to  verbalize  this 
knowledge  to  learners  through  behavioral  role  modeling  and  documentation. 
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Table  1.  Seminal  Research  Articles  in  Supporting  Research  Areas 


Research  Area 

Citation(s) 

Performance  Appraisal 

Brett  &  Atwater  (2001);  Borman  (1997) 

Instructional 

Technology 

Baird,  Holland,  &  Deacon  (1999);  Briggs  (1977) 

Human  Learning 

Garvin  (2000);  Bilodeau  (1966);  Atkinson  &  Shiffrin  (1968) 

Task  Training 

Holding  (1965);  Patrick  (1992);  Goldstein  &  Ford  (1992) 

Team  Performance 

Minionis,  Zaccaro,  &  Perez  (1995);  McIntyre  &  Salas  (1995) 

Leadership 

Hoyt,  Halverson,  Murphy  &  Watson  (2003) 

Feedback  Design  and 
Implementation 

Nadler  (1979);  Downs,  Johnson,  &  Fallesen  (1987);  Ammons 
(1956) 

Knowledge 

Management 

Nonaka  &  Takeuchi  (1995);  Joshi,  K.D.,  Sarker,  S.,  &  Sarker, 

S.  (2007) 

Recommendations  were  also  presented  by  Newlin  and  Bliss  (2007)  that  are 
relevant  for  the  training  audience: 

•  Trainees  who  go  through  joint  AARs  must  be  encouraged  to  work  collaboratively 
with  other  team  members  to  achieve  performance  goals  and  learn  from  the  AAR 
process. 

•  Team  members  can  acquire  skills  and  build  mental  representations  by  acting  out 
behavioral  responses  and  understanding  the  theoretical  benefit  of  doing  so. 

•  Past  research  suggests  that  groups  perform  more  efficiently  and  appropriately 
when  team  members  share  mental  models.  Therefore,  AAR  sessions  should  be 
designed  to  yield  a  common  understanding  of  the  trained  material. 

•  AARs  should  promote  shared  mental  models  by  encouraging  team  members  to 
actively  participate  in  discussions. 

•  When  new  team  members  enter  an  AAR  group,  experienced  trainees  should  share 
mental  model  information  quickly  to  assist  the  new  member  to  form  an  accurate 
mental  representation. 

•  AAR  sessions  should  focus  on  both  individual  and  group  feedback. 

•  AARs  should  focus  on  unconventional  or  novel  measures  of  group  effectiveness, 
including  measures  of  cohesion  and  cooperation. 

Generalized  Research  Findings  by  Domain  -  Human  Cognition 

The  first  area  that  we  consider  to  be  central  to  AAR  effectiveness  is  human 
(trainee)  cognition.  It  is  important  to  emphasize  here  that  almost  all  functions  and 
elements  of  human  cognition  bear  some  relevance  to  the  design  and  implementation  of 
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AARs.  For  that  reason,  an  unabridged  discussion  of  human  cognition  as  it  relates  to 
AAR  is  a  daunting  task.  We  have  constrained  our  treatment  to  those  elements  that  we 
believe  are  most  directly  concerned  with  the  facilitation  of  trainee  learning  and 
performance. 

Cognitive  learning.  The  AAR  process  deals  heavily  with  cognitive  learning, 
because  such  learning  will  be  continuous  when  Soldiers  solve  problems  on  the  battlefield 
and  away  from  it.  During  battle,  there  is  an  abundant  amount  of  infonnation;  to 
successfully  perform  their  role,  Soldiers  must  learn  quickly  what  to  expect  from  the  battle 
situation  and  how  to  manage  available  resources  to  make  effective  decisions.  During  the 
AAR,  facilitating  cognitive  learning  is  crucial,  as  the  participants  will  be  expected  to  take 
the  infonnation  and  incorporate  it  into  their  current  existing  knowledge  framework.  If 
accomplished  effectively,  trainees  will  be  able  to  form  a  meaningful  network  of 
information  that  transfers  to  operational  situations  when  needed. 

Research  that  pertains  to  cognitive  learning  theory  concerns  mainly  the  transfer  of 
information  between  short  term  and  long  term  memory  storage  areas.  Long  tenn  storage 
is  a  collection  of  past  information  learned,  semantic  meanings,  episodic  memories,  and 
the  linkages  among  items  in  long  term  storage.  Short  tenn  storage  is  an  intermediate 
“buffer”  that  contains  information  of  events  that  have  just  recently  happened,  and 
transferring  these  items  from  short  tenn  to  long  term  storage  is  the  main  aspect  of  the 
cognitive  learning  process  (Atkinson  &  Shiffrin,  1968). 

For  many  years,  training  researchers  have  encouraged  the  development  of  training 
procedures  to  facilitate  the  transfer  of  infonnation  from  short  term  to  long  term  storage. 
Specific  foci  of  these  procedures  have  included  utilizing  methods  and  techniques  to  speed 
up  the  learning  process,  ensure  long  term  access,  and  provide  many  meaningful  cues 
leading  to  the  information  stored  (Patrick,  1992).  The  learning  process  consists  of 
encoding  and  retrieval  of  learned  infonnation.  Encoding,  in  particular,  concerns  the 
process  of  placing  the  information  in  LTS,  while  retrieval  deals  with  the  process  of 
accessing  this  stored  information  (Hunt  &  Ellis,  2004). 

Learned  knowledge  is  maintained  in  long  term  storage  as  a  “semantic  network”  or 
schema.  These  terms  basically  suggest  a  cluster  of  related  items  that  become  activated 
when  one  single  entity  in  the  network  is  activated  (Crestani,  1997).  For  example,  if  the 
schema  contains  infonnation  of  a  classroom,  once  “desk”  is  cued,  “teacher”  or  “lecture” 
will  be  available  for  retrieval  faster  than  it  would  be  without  cueing  the  schema.  Any 
given  item’s  connections  can  be  strengthened  with  frequent  and  consistent  pairings,  as 
well  as  meaningful,  self-generated  cues.  Deep  processing  and  self-generation  of  cues 
lead  to  a  stronger  bond  between  items  in  memory  (Hunt  &  Ellis,  2004).  The  links 
between  items  in  memory  are  not  immune  to  decay,  as  the  links  leading  to  information 
can  weaken  if  not  used  on  a  somewhat  regular  basis.  Forgetting  stored  information  can 
occur  when  there  is  little  frequency  or  recency  of  the  retrieval  process,  interfering 
associations,  or  little  association  with  other  infonnation.  Longer  spans  of  time  between 
recall  of  the  items  will  lead  to  a  weakening  of  the  associations,  allow  for  the  related  cues 
to  be  associated  with  other  items  (Wickens,  Lee,  Liu,  &  Becker,  2004). 

Implications.  There  are  implications  of  such  “spreading  activation”  for  training. 
Specifically,  the  theory  suggests  that  for  training  to  be  successful,  attempts  should  be 
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made  to  encourage  deep,  meaningful  processing,  which  ensures  multiple  cues  and 
stronger  associations  between  items  in  memory.  This  starkly  contrasts  with  simple  rote 
memorization,  as  the  infonnation  stored  will  be  difficult  to  access  without  the  meaningful 
associations  or  links  leading  to  the  stored  knowledge.  Attempts  should  be  made  to  ensure 
that  the  information  links  are  consistently  paired  with  the  events  or  stored  knowledge  and 
accessed  often,  as  disuse  will  lead  to  information  decay,  or  forgetting.  In  the  context  of 
an  AAR,  review  of  missions  completed  should  be  approached  from  multiple  viewpoints 
(for  example,  command,  OPFOR,  teamed  blue  force,  communications,  and  support)  to 
enable  complete  processing  of  the  material  for  later  recall  and  application  (see  Paivio, 
1971). 

Memory.  Knowledge  stored  in  long  term  storage  may  be  categorized  four  ways: 
schemas,  scripts,  mental  models,  and  cognitive  maps  (Doyle  &  Ford,  1998).  A  schema  is 
a  collection  of  semantically  related  items  in  memory,  such  as  “classroom”,  while  a  script 
is  an  extension  of  this  concept,  adding  in  a  time  sensitivity  or  process  to  this  information 
network.  Scripts  are  episodic  in  nature,  such  as  “steps  involved  in  conducting  an  AAR” 
(Schank  &  Abelson  1977).  Mental  models  represent  learners’  understandings  of  systems 
or  equipment,  how  they  are  used,  when  they  are  used,  and  why.  Cognitive  maps  are 
generally  a  map  layout  or  spatial  infonnation.  Because  of  the  complexity  of  AARs  in 
practice,  it  is  likely  that  trainees  will  often  be  called  on  to  form  schemas  (for  example 
“ways  to  identify  OPFOR  members”),  scripts  (“potential  ways  that  identified  OPFOR 
members  may  disguise  themselves  when  conducting  espionage”),  mental  models  (“how 
can  I  use  robotic  tools  to  search  for  insurgents?”)  and  cognitive  maps  (“where  have  we 
already  searched  for  insurgents  in  this  community?”). 

Episodic  infonnation  is  used  to  catalogue  a  series  of  events.  It  is  important  to 
understand  the  many  aspects  of  episodic  infonnation,  as  research  has  demonstrated  a 
tendency  for  learners  to  remember  information  falsely  under  stressful  or  overloaded 
conditions  (Wright  &  Davies,  2007).  Episodic  memory  can  be  distorted  or  modified  by 
the  retriever  in  three  different  stages.  Encoding  inconectly  can  lead  to  a  biased  episodic 
memory,  because  there  is  a  limit  on  attention  capacity  at  any  given  time.  It  can  be 
expected  that  items  demanding  more  attention  (a  dangerous  item,  such  as  a  weapon)  will 
detract  from  the  person’s  ability  to  encode  the  rest  of  the  scene  correctly.  Secondly,  a 
storage  bias  can  occur  to  alter  the  integrity  of  episodic  memory.  Cognitive  researchers 
have  demonstrated  the  ubiquity  of  storage  bias  as  the  mind  attempts  to  “fill  in  the  blanks” 
of  memory  via  top  down  processing  (Hyman,  1999).  Missing  infonnation  is  often  filled 
in  with  items  that  are  expected  to  potentially  be  there,  often  the  items  activated  in  the 
schema  of  the  event.  For  example,  during  an  AAR,  a  trainee  might  be  inclined  to 
remember  there  being  more  OPFOR  present  on  the  battlefield  than  were  actually  there. 

A  related  problem  involves  retrieval  biases  that  can  occur  when  infonnation  is  not 
retrieved  (because  of  a  lack  of  conceptual  links),  or  retrieved  incorrectly. 

Implications.  As  suggested  above,  false  remembrance  can  impact  learning  that 
transpires  in  a  training  situation,  as  information  remembered  incorrectly  can  directly 
influence  the  acceptance  (or  rejection)  of  perceived  feedback.  Feedback  that  misaligns 
with  a  person’s  perception  of  their  own  performance  will  be  rejected  by  that  person,  so  it 
is  crucial  that  if  episodic  memory  of  a  battle  situation  reflects  false  memory 
reconstruction,  AARs  take  action  to  present  (where  possible)  ground  truth  in  a 
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convincing  manner.  The  use  of  automated  tools  for  AAR  [such  as  the  Dismounted 
Infantry  Virtual  After  Action  Review  System  (DIVAARS;  Clark,  Lamp  ton,  Martin,  & 
Bliss,  2004)]  may  provide  an  excellent  way  to  capture  training  action  for  subsequent 
replay.  Furthermore,  facilitators  should  be  trained  to  recognize  the  potential  for  false 
memories,  and  taught  strategies  by  which  they  may  be  prevented  or  corrected.  Generally, 
the  AAR  facilitator  should  ensure  that  the  best  possible  representation  of  the  battle  should 
be  recreated,  and  that  each  person  accepts  this  representation  as  what  actually  happened. 

Situation  awareness.  Another  cognitive  phenomenon  that  may  influence  the 
success  of  an  AAR  session  is  situation  awareness.  Defined  formally  by  Mica  Endsley 
(1995),  situation  awareness  refers  to  a  person’s  general  understanding  of  the  real  time 
changes  in  their  surrounding  environment.  This  includes  perceiving  their  environment, 
understanding  how  the  elements  work  together,  and  predicting  how  these  elements  will 
interact  in  the  near  future  (termed  “levels  1,  2,  and  3”  of  situation  awareness  (Durso 
&  Gronlund,  1999).  Situational  awareness  also  includes  the  ability  to  “filter  out” 
information  that  is  irrelevant  to  the  task  at  hand  (Endsley,  1995).  An  AAR  facilitator 
should  be  committed  to  addressing  learners’  shortcomings  or  proficiencies  in  situational 
awareness,  and  encourage  knowledge  transfer  to  ensure  that  each  participant  will 
understand  which  information  is  crucial  to  battle  success  and  which  information  is 
irrelevant.  A  variety  of  tools  have  been  proposed  to  enable  the  measurement  of  situation 
awareness,  including  questionnaires,  task  probes,  and  secondary  tasks.  Yet,  in  most  AAR 
situations,  little  if  any  systematic  attempt  is  ever  made  to  determine  the  situation 
awareness  that  trainees  demonstrate.  Recent  work  regarding  the  concept  of  “teamed 
situation  awareness”  (Endsley  &  Robinson,  1996)  may  be  particularly  relevant  here,  as  a 
primary  goal  of  AARs  should  be  the  facilitation  of  shared  awareness  by  team  members. 

Implications.  Situation  awareness  often  has  a  direct  impact  on  problem  solving 
and  decision  making  strategies.  AAR  feedback  should  assist  trainees  to  improve  their 
skills  for  these  things.  Problem  solving  involves  taking  cognitive  steps  to  arrive  at  a 
desired  “goal  state”  or  solution.  Troubleshooting  adds  a  step  to  this  process,  as  cognitive 
steps  to  diagnose  a  particular  problem,  followed  by  the  problem  solving  step  (Wickens  et 
ah,  2004).  Working  memory,  or  one’s  ability  to  encode  infonnation  in  LTS  while  doing 
cognitively  demanding  processes  simultaneously,  imposes  limits  to  a  person’s  ability  to 
troubleshoot  and  problem  solve.  AAR  facilitators  should  work  to  identify  and  engender 
this  ability  in  those  that  seem  proficient  at  it,  as  well  as  ensuring  that  the  “problem 
solver”  is  able  to  walk  the  other  AAR  participants  through  his  or  her  steps  to  arrive  at  the 
goal  state. 

Attention.  Though  much  of  our  discussion  has  been  targeted  toward  the  AAR 
facilitator,  it  is  important  to  recognize  that  individual  trainees  bear  a  considerable 
responsibility  for  cognitive  learning.  Personal  effort  is  crucial  to  ensure  learning  and 
retention,  because  combat  situations  frequently  present  difficult  or  complex  concepts  that 
can  be  judged  by  trainees  as  too  much  effort  to  be  worth  their  while.  People  in  general 
often  pursue  strategies  that  utilize  low-effort  actions  rather  than  more  complex  (but  more 
efficient)  high-effort  strategies.  Economic  or  “rational”  theories  of  decision  making 
(Sternberg  &  Smith,  1988)  often  stress  the  potential  for  individuals  to  engage  in  a  sort  of 
“cost-benefit  analysis”  to  determine  whether  a  course  of  action  is  worth  the  effort.  In 
certain  combat  situations,  individual  trainees  may  judge  information  or  knowledge  to  be 
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gained  as  trivial  compared  to  the  effort  required  to  gain  it  (Fennema  &  Kleinmuntz, 

1995).  An  example  relevant  to  the  AAR  process  would  be  a  trainee  in  one  particular  role 
being  uninterested  or  “tuning  out”  during  the  discussion  of  another  team  member’s 
actions.  The  AAR  benefits  can  be  weakened  by  this  behavior  if  trainees  do  not 
understand  the  knowledge  that  accrues  from  shared  mental  models.  AAR  facilitators, 
then,  should  work  to  enable  the  sharing  of  crucial  knowledge,  so  that  audience  members 
see  its  value  and  are  not  tempted  to  consider  the  cost  as  excessive  relative  to  the  benefit. 
One  possible  method  for  achieving  this  is  to  reword  lessons  in  tenns  of  shared  benefit; 
another  might  be  to  have  team  members  verbalize  the  benefit  that  each  of  them  has 
gained. 

As  a  cognitive  resource,  attention  is  essential  for  learning  to  take  place.  Scientists 
have  for  many  years  considered  the  abilities  of  learners  to  divide  their  attention  between 
multiple  demanding  sources.  As  with  memory  and  decision  making,  cognitive  resources 
are  important  to  understanding  why  some  individuals  might  have  difficulty  timesharing 
attention  during  an  exercise.  Four  major  factors  contribute  to  the  success  (or  failure)  in 
dividing  one’s  attention:  resource  demand,  structure,  similarity,  and  resource  allocation. 
Resource  demand  deals  with  how  much  cognitive  effort  is  required  by  a  task.  As  the 
demand  of  one  or  more  shared  tasks  increase,  performance  on  all  tasks  will  decrease. 
Structure  refers  to  which  resource  the  task  depletes,  such  as  visual  processing,  auditory 
processing  or  spatial  processing  (Wickens,  2002).  Similarity  among  tasks  is  often 
discussed  with  regard  to  structure,  but  deals  more  with  the  cognitive  resources  each  task 
demands.  Finally,  resource  allocation  deals  with  attention,  and  how  much  is  available  for 
each  respective  task. 

Implications.  The  role  of  an  AAR  should  be  to  identify  these  situations  that 
require  divided  attention,  so  that  trainees  may  recognize  them  and  disperse  the 
responsibility  of  the  task  to  other  individuals  with  lower  task  demands  accordingly. 
Multiple  tasks  that  require  too  much  attention  or  resources  will  not  be  performed  well  by 
individual  trainees  (Wickens,  2002).  AAR  facilitators  should  be  keen  to  identify  persons 
who  are  vulnerable  to  such  strains,  and  encourage  other  group  members  to  assume  task 
responsibilities  when  needed.  Ultimately  the  overall  task  will  benefit  from  enhanced 
performance  and  flexibility  for  additional  cognitive  challenges  as  they  arise. 

Expertise.  With  repeated  exposure  to  training,  Soldiers  are  expected  to  develop 
“expertise”  in  their  particular  roles.  The  study  of  expertise  development  has  a  long  and 
important  history  (Ericsson,  1996).  As  trainees  progress  in  expertise,  they  typically  go 
through  a  number  of  stages  of  cognitive  skill  development.  Initially,  knowledge  about  a 
task  is  represented  as  “declarative  knowledge”,  or  knowledge  that  is  represented  loosely 
as  a  set  of  definitions  that  lacks  practical  application.  After  repeated  practice,  declarative 
knowledge  gives  way  to  “procedural  knowledge”,  which  refers  to  hands-on  knowledge  of 
the  task.  Procedural  knowledge  represents  the  development  of  explicit  rules  to  govern 
under  specific  occurrences.  For  example,  a  trainee  may  learn  a  procedure  for  building 
assault  that  represents  a  string  of  behaviors  (approach  building  under  cover,  enter 
building  from  rear,  search  perimeter,  and  so  forth).  Automaticity  develops  after  many 
iterations  of  performing  the  activity;  at  an  automatic  level,  individuals  have  more 
efficient  ways  of  chunking  infonnation.  Little  attention  is  required  to  perform  the  task, 
and  it  is  carried  out  quickly  and  efficiently  (Hunt  &  Ellis,  2004).  At  the  automatic  level, 
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information  garnered  during  an  exercise  is  likely  to  be  stored  and  retrieved  more  quickly 
and  efficiently  than  at  prior  levels. 

Implications.  The  development  of  expertise  is  something  the  AAR  should 
heavily  focus  on  and  facilitate.  Although  practice  is  crucial  to  developing  expertise,  any 
steps  that  can  be  taken  to  shorten  the  amount  of  time  needed  to  gain  expertise  should  be 
emphasized.  A  variety  of  training  methods  have  been  proposed  in  the  literature  (see 
Patrick,  1992):  adaptive  training,  scaffolding,  part-task  training,  and  distributed  training 
are  all  viable  options  and  should  be  weighed  to  detennine  their  appropriateness  for 
various  training  situations.  Expertise  development  is  crucial,  as  it  will  quicken  the 
process  of  Soldiers  developing  skills  and  efficiency  they  need  to  be  successful  when 
performing  their  tasks. 

The  AAR  facilitator  should  work  to  ensure  that  individuals  who  are  “quicker”  at 
acquiring  expertise  verbalize  the  processes  and  methods  they  use  so  that  members  who 
have  not  achieved  expert  status  can  benefit  from  that  knowledge.  Each  AAR  session 
should  have  information  to  strengthen  the  mental  models  of  the  battlefield  system,  so  that 
information  is  recalled  more  readily  in  a  time  of  need. 

Facilitators  must  be  mindful  that  one  of  the  challenges  to  using  “self  report”  by 
unit  experts  to  impart  successful  processes  and  methods  to  teammates  is  the  great 
potential  that  the  resulting  information  often  contains  significant  errors  and  omissions. 
These  errors  are  not  often  recognized  by  participants  who  solved  important  problems  in 
emergency  situations  and  wish  to  give  accurate  reports  on  their  solutions  because  the 
knowledge  they  are  describing  is  largely  automated  and  unconscious.  As  AAR  can  be 
used  as  a  process  to  document  new  methods  to  overcome  unique  challenges  this 
phenomenon  can  be  problematic  if  incomplete  or  inaccurate  lessons  learned  are  passed 
from  experts  to  novice  participants.  There  is  some  evidence  to  support  the  belief  that  one 
needs  to  use  methods  such  as  Cognitive  Task  Analyses  (CTA)  in  conjunction  with  AAR 
to  actually  obtain  a  high  degree  of  accuracy  when  documenting  expert  solutions  to 
complex  problems.  (Clark,  2007) 

Generalized  Research  Findings  by  Domain  -  Team  Dynamics 

A  team  is  defined  as  a  small  number  of  people  who  hold  specialized  or  individual 
roles  or  functions,  and  act  dynamically  towards  a  shared  goal  for  which  each  member 
views  themselves  as  mutually  accountable.  One  of  the  defining  aspects  of  a  team  is 
interdependence  (McIntyre  &  Salas,  1995).  Teams  are  viewed  as  a  “work  unit”  by  both 
members  and  non-members  alike.  The  goals  held  by  the  team  reinforce  the  notion  of 
interdependence  among  the  group  members.  Each  member  of  the  group  carries  out  a 
different  role,  and  the  output  or  outcome  created  by  the  group  is  a  combination  of  all  the 
member's  efforts. 

An  interesting  component  of  military  teams  is  that  the  roles  of  the  team  members 
are  often  prescribed  by  rank,  experience,  and  convention.  However,  in  the  event  that  a 
person  who  is  designated  as  the  leader  becomes  incapacitated,  it  may  be  necessary  for 
other  team  members  to  adopt  aspects  of  that  role.  An  important  element  of  an  AAR  may 
be  to  rehearse  contingencies  in  such  events.  Doing  so  may  allow  the  team  members  to 
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gain  a  broader  appreciation  of  each  others’  roles  and  may  ultimately  result  in  better 
training. 

Communication.  Successful  team  perfonnance  depends  on  the  right  mix  of 
members  who  understand  their  own  roles  as  well  as  other  members’  roles  (Belbin,  1981). 
The  team  leader  should  have  a  style  that  matches  the  tone  and  nature  of  the  mission,  and 
the  other  team  individuals  should  have  the  correct  complementary  task  and  teamwork 
skills  to  be  successful.  Communication  among  team  members  is  also  crucial  to  success, 
so  much  so  that  the  size  of  the  team  is  limited  by  the  ability  of  the  members  to  quickly 
and  accurately  communicate  with  each  other.  If  members  of  the  unit  face  issues 
communicating  among  themselves,  the  group  should  be  broken  down  for  training 
purposes  to  facilitate  the  dissemination  of  information. 

Implications.  As  communication  has  been  proven  to  be  an  essential  part  of 
teamwork  success,  one  of  the  AAR’s  main  foci  should  be  facilitating  effective,  effortless, 
and  efficient  communication  among  its  participants.  Special  attention  and  care  should  be 
given  to  communication  errors  or  breakdowns,  and  steps  to  avoid  similar  situations 
should  be  explored,  because  communication  is  central  to  team  performance.  An  example 
of  the  fragility  of  team  communication  in  AAR  situations  is  shown  by  Gratch  and  Mao 
(2003),  who  investigate  team  communication  factors  such  as  attribution  and  blaming 
within  team  interaction. 

Goals.  Team  effectiveness  is  detennined  by  several  key  factors  of  team 
interaction.  Within  all  group  members,  a  common,  meaningful  purpose  needs  to  exist. 
The  product  or  output  of  the  team  should  be  well  defined,  so  that  each  member  knows 
exactly  what  the  objectives  of  the  team  are.  Setting  specific  perfonnance  goals,  rather 
than  general  ones,  for  both  the  team  overall  and  each  member  will  lead  to  greater  team 
effectiveness.  Each  member  of  the  team  should  be  committed  to  working  as  a  team, 
maintaining  mutual  dependent  needs  on  the  other  members'  abilities.  Team  effectiveness 
hinges  upon  the  leader  following  the  project  guidelines,  and  appropriately  delegating 
responsibilities  to  the  correct  team  member.  The  execution  of  these  responsibilities  needs 
to  be  well  coordinated  with  the  other's  responsibilities,  and  each  team  member  needs  to 
feel  a  shared  accountability  towards  the  overall  performance  of  the  group. 

Implications.  Goal  driven  interaction  between  team  members  should  be 
encouraged  and  engendered  within  the  AAR.  For  many  years,  behavioral  scientists  have 
acknowledged  the  importance  of  goal  setting  in  the  perfonnance  appraisal  process 
(DeShon,  Kozlowski,  Schmidt,  Milner,  &  Wiechmann,  2004).  Any  verbalizations  made 
by  trainees  during  the  AAR  session,  particularly  those  referring  to  a  specific  team  goal, 
are  signs  of  team  growth  and  should  be  encouraged.  Group  accountability  and 
interdependence  should  also  be  emphasized,  and  feedback  directed  at  the  group  should  be 
directed  at  strengthening  these  concepts  for  the  group  (Hackman,  1987). 

Team  dynamics.  Adherence  to  these  teamwork  tenants  leads  to  optimal 
performance;  however,  there  are  some  team  dynamics  known  to  cause  poor  performance. 
Disagreements  over  authority  or  perceived  power  can  cause  a  disconnect  within  the 
groups’  shared  goals.  Members  who  maintain  incongruent  nonns  or  values  can  lead  poor 
performance,  as  well  as  low  team  morale,  undefined  team  roles  or  structure,  a  lack  of 
mutual  goals,  poor  communication,  and  finally,  lack  of  feedback  or  criticism.  Also, 
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rewarding  individual  performance  can  cause  a  decrease  in  commitment  to  team  goals, 
which  decreases  team  effectiveness.  Facilitators  of  the  AAR  should  pay  special  attention 
to  these  team  shortcomings,  and  offer  up  corrective  feedback  to  the  group  to  ensure  these 
characteristics  are  lessened  or  abandoned  (Nadler,  1979). 

Team  training.  Team  training  is  crucial  to  the  eventual  perfonnance  of  the  team. 
Properly  structured  advanced  training  sessions  can  ensure  that  the  members  of  the  team 
can  work  together  and  communicate  more  effectively  before  real-time  performance  of  the 
task.  Building  “task  work”  skills  (tasks  performed  by  each  individual  in  the  team  task) 
trained  without  the  other  team  members  will  undermine  the  development  of  “teamwork 
skills”.  Teamwork  skills  include  cooperation,  coordination,  communication,  adaptability, 
ability  to  give  criticism,  acceptance  of  criticism,  and  maintaining  high  team  morale. 
Groups  that  practice  their  individual  tasks  concurrently  with  the  other  members  will 
facilitate  an  increase  in  teamwork  skills,  while  those  that  practice  them  individually  will 
not  perform  as  well  in  a  team  setting.  Team  building  activities  that  focus  on  goal  setting, 
problem  solving,  and  interpersonal  relationships  do  not  do  as  well  as  those  that  focus  on 
roles  and  responsibility  development  of  each  team  member  (McIntyre  &  Salas,  1995). 

Implications.  In  training  teams,  cross  training  is  considered  a  highly  effective 
practice.  In  cross  training,  team  members  are  trained  to  do  other  members’  jobs.  This 
can  lead  to  higher  team  cohesiveness  and  more  efficient  communication,  as  members  are 
expected  to  have  an  intimate  understanding  of  what  the  other  members  need  to  know  and 
when. 

Military  combat  teams  have  been  characterized  as  "action/negotiation  teams", 
which  require  high  role  differentiation  (and  the  team  life  spans  are  often  long),  and  high 
synchronization  with  outside  units.  The  teams  are  guided  by  externally  imposed  pacing 
and  cycles  of  work  that  are  often  short  and  take  place  under  dynamic  conditions. 

Training  and  technology  are  essential  for  determining  team  effectiveness. 

Dynamic  and  complex  environments  can  lead  to  high  stress  work  tasks  that  can 
cause  cognitive  overload.  Team  communication  and  performance  decreases  as  a  result. 
Training  must  combat  this  effect  by  facilitating  the  development  of  shared  mental 
models,  effective  communication,  stress  adaptation,  maintenance  of  situation  awareness, 
and  coordination.  Shared  mental  models  combat  the  effects  of  stress  by  allowing  for 
anticipatory  communication  (or  a  lowering  of  the  need  for  communication),  such  as  a 
person  offering  crucial  information  before  the  information  is  requested.  Good  teams  take 
advantage  of  the  time  between  these  high-stress  scenarios,  at  which  point  they  take  the 
opportunity  to  share  infonnation  and  build  their  shared  mental  model  for  the  subsequent 
encounters.  Effective  teams  often  distribute  work  load  beyond  what  is  expected  in  times 
of  high  demand,  blurring  the  lines  between  team  members  in  stressful  situations. 

Generalized  Research  Findings  by  Domain  -  Feedback 

Feedback  is  the  process  of  providing  an  individual  or  group  information  about 
their  performance.  The  feedback  research  literature  generally  supports  the  idea  that 
providing  an  individual  with  information  about  perfonnance  increases  motivation  and 
improves  subsequent  performances  (London,  2003).  Feedback  consists  of  a  source,  a 
message,  and  a  recipient,  all  of  which  play  special  roles  in  the  feedback  process. 
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Delivering  feedback.  The  major  difference  between  a  critique  and  an  AAR  is 
how  feedback  is  acquired  by  the  trainee.  In  a  critique,  an  evaluator  provides  feedback  to 
the  trainee  in  the  fonn  of  statements  about  good  and  poor  performance.  In  an  AAR,  the 
facilitator  provides  feedback  to  the  trainee  in  such  a  way  that  the  trainee  identifies  their 
own  good  and  poor  performance.  The  advantage  of  the  trainee  forming  their  own 
feedback  is  that  there  is  often  greater  acceptance  of  self-detennined  feedback  than  that 
which  is  presented  from  others.  Self-feedback  and  task-feedback  are  generally  regarded 
as  more  reliable  by  the  recipient,  and  as  such  should  not  be  ignored  (Ilgen,  Fisher 
&Taylor  1979). 

The  source  of  feedback  can  be  broken  down  into  three  categories;  individuals  who 
have  observed  the  task  performance,  the  task  environment  itself  (objects  in  the 
environment  displaying  an  effect  when  you  act  upon  them),  and  finally  the  recipients 
themselves.  Feedback  in  the  AAR  mainly  consists  of  outside  individuals  who  have 
observed  task  performance  and  present  this  information  to  trainees  to  help  them  to 
understand  their  performance.  Ideally,  trainees  use  this  information,  along  with  task 
feedback,  to  form  self-feedback.  The  closer  the  source  of  the  feedback  is  to  the  recipient, 
the  more  likely  they  will  accept  and  respond  to  it  (Greller  &  Herold,  1975). 

Individuals  providing  feedback  can  differ  in  their  power  and  credibility  (which 
are  independent  of  each  other),  each  having  their  own  effects  on  the  reception  and 
acceptance  of  the  feedback.  Credible  sources  will  have  their  feedback  more  readily 
accepted  and  acted  upon,  while  non-credible  (at  least  perceived)  sources  will  have  their 
feedback  disregarded  or  ignored.  Non-credible  sources’  feedback  is  perceived  as  faulty, 
and  is  thought  to  not  actually  refer  to  the  receiver’s  performance.  The  less  the  recipient 
feels  the  feedback  deals  with  their  actual  performance,  the  less  they’ll  use  the  infonnation 
to  alter  their  behavior.  Sources  with  more  power  are  perceived  as  being  important,  and 
will  have  their  feedback  attended  despite  the  feelings  of  the  receiver. 

Implications.  The  feedback  given  in  the  AAR  should  incorporate  these  concepts. 
Feedback  given  should  be  delivered  so  that  the  recipient s)  feel  it  actually  reflects  their 
performance,  which  can  present  a  problem  when  dealing  with  multiple  people.  Great 
care  should  be  given  to  ensuring  each  AAR  participant  agrees  with  the  assessment  of 
their  perfonnance,  otherwise  they  will  simply  disregard  the  information.  The  facilitator 
should  be  highly  concerned  with  his  or  her  credibility,  and  take  steps  to  ensure  that  this 
credibility  stays  as  intact  as  it  possibly  can  be. 

The  message  of  the  feedback  contains  some  form  of  information  about  the 
receiver’s  past  information  (Annet,  1969),  which  should  provide  some  sense  of  accuracy, 
correctness,  and  adequacy  of  the  response  (Bourne,  1966).  The  receiver  should  be  able  to 
convert  the  feedback  message  into  meaningful  units,  as  ambiguous  information  will  have 
little  effect  on  behavior  (Annet,  1969).  The  infonnation  value  of  the  message  depends  on 
the  amount  of  knowledge  of  the  recipient’s  performance  the  feedback  provides  above  and 
beyond  what  they  already  know  about  their  perfonnance.  The  cybernetic  approach  to 
feedback  suggests  it  is  most  effective  when  it  causes  an  increase  in  knowledge  through  a 
reduction  of  uncertainty.  This  is  done  by  eliminating  half  of  the  alternative  explanations 
for  behavior  (Shannon  &  Weaver,  1949). 
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AAR  practices  should  be  sure  to  provide  feedback  information  in  a  manner  that  is 
easily  broken  down  into  meaningful  units  for  the  receivers  of  the  feedback.  The  more 
specific  the  information  given,  the  more  likely  the  receiver  will  be  able  to  incorporate  the 
feedback  and  use  it  in  later  performances.  The  information  provided  should  contain 
information  that  subsidizes  the  AAR  participant’s  existing  knowledge  of  team  or 
individual  performance.  Dispensed  information  should  also  reduce  the  participants’ 
uncertainty  of  the  outcomes  of  particular  events.  They  should  be  clear  on  who’s  actions 
led  to  what  event,  as  reducing  the  ambiguity  of  the  situation  will  provide  the  ability  to 
alter  behavior. 

There  are  four  levels  of  recipient's  processing  of  feedback:  perception  of  the 
feedback,  acceptance  of  the  feedback,  desire  to  respond  to  the  feedback,  and  the  intended 
response  (Ilgen,  Fisher  &Taylor,  1979).  Each  of  these  stages  represents  a  distinct 
moment  in  the  AAR  feedback  process,  with  each  stage  providing  its  own  obstacles  to  the 
desired  outcome.  Each  stage  also  can  be  affected  by  the  previously  discussed  aspects  of 
feedback,  the  source,  the  message,  and  the  recipient. 

Perception  of  feedback.  The  perception  of  the  feedback  refers  to  a  person's 
receiving  and  interpreting  the  feedback,  and  deals  with  how  accurately  this  process 
occurs.  Feedback,  to  have  an  effect  on  behavior,  must  be  believed  and  understood  by  the 
receiver.  The  source  of  the  feedback  interacts  with  the  perception  of  the  feedback,  as  self 
and  task  feedback  are  remembered  more  accurately  than  supervisor  feedback. 

The  message  can  alter  the  perception  of  feedback  in  three  ways;  the  timing,  the 
sign,  and  the  frequency.  Timing  deals  with  the  speed  in  which  the  information  is 
delivered  after  the  task  is  complete.  The  quicker  the  information  gets  to  the  recipients, 
the  more  likely  they  are  to  attribute  their  actual  performance  with  the  feedback  they 
receive  (Ammons,  1956).  The  longer  the  amount  of  time,  the  chances  of  linking  the  two 
lessens.  The  next  aspect  of  the  message  is  its  sign.  Feedback  with  a  positive  sign  is 
feedback  that  confirms  or  supports  correct  task  performance.  Positive  feedback  is 
received  more  readily  and  accurately  than  negative  feedback  (Halperin  et  ah,  1976). 
Negative  feedback  is  feedback  that  informs  the  receiver  that  the  task  was  performed 
incorrectly  or  not  well  enough.  Retention  of  negative  feedback  has  been  shown  to  be 
faulty  over  time,  as  the  receiver  faces  a  challenge  accepting  these  negative  attributions 
about  themselves.  Lastly,  the  message  can  differ  in  its  frequency,  with  more  frequent 
feedback  leading  to  a  bigger  effect  on  perfonnance. 

Finally,  the  recipients  themselves  can  detennine  the  perception  of  the  feedback. 
The  individual  holds  several  expectations  about  their  performance,  and  when  the 
feedback  does  not  meet  these  expectations,  the  feedback's  credibility  is  called  into 
question.  Persons  with  external  loci  of  control  are  more  likely  to  be  motivated  by 
external  source  feedback,  while  persons  with  internal  loci  of  control  are  more  motivated 
by  self-discovery  feedback  (task-supplied)  (Baron,  Cowan,  &  Ganz,  1974).  Also,  high 
self-esteem  individuals  are  more  likely  to  rely  on  themselves  for  feedback,  and  raise  their 
competency  more  when  positive  feedback  is  received. 

Acceptance  of  feedback.  The  acceptance  of  feedback,  particularly  external 
feedback,  involves  the  recipients  belief  that  the  feedback  actually  refers  to  their 
performance.  After  the  feedback  has  actually  been  perceived,  the  feedback  must  then  be 
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accepted  by  the  participant.  If  there  is  any  disconnect  between  the  facilitators  view  of  the 
exercise  and  the  trainee's  view  of  the  exercise,  the  effect  of  feedback  will  be  reduced. 

The  source  of  the  feedback  plays  a  large  role  in  acceptance  of  the  feedback,  with 
self-feedback  and  more  credible  sources  of  external  feedback  having  their  feedback 
accepted.  Five  characteristics  of  the  external  source  have  been  identified  to  play  a  role  in 
feedback  acceptance  by  Giffin  (1967);  Expertise,  reliability,  intention  toward  the  listener, 
dynamism,  and  personal  attraction.  Expertise  refers  to  the  source's  knowledge  of  the 
task  at  hand,  with  more  perceived  knowledge  leading  to  more  acceptance  (Tuckman  & 
Oliver,  1968).  Reliability  refers  to  the  source's  tendency  to  be  consistent  in  their  methods, 
delivery,  judgments  contained  in  their  feedback,  and  how  often  the  receiver  accepts  their 
feedback  (Giffin,  1967).  Intention  towards  the  receiver  is  how  the  AAR  participant  feels 
about  the  source's  motives  for  giving  feedback.  If  the  participant  feels  that  the  facilitator 
is  “on  their  side”,  they  will  be  more  likely  to  accept  the  feedback  (Hogan,  Fisher,  & 
Morrison,  1974).  Dynamism  is  how  energetic  or  bold  the  source  is,  with  more  energy 
leading  to  a  higher  acceptance  of  feedback.  Personal  attraction  is  how  the  receiver  feels 
about  the  source  (whether  the  source  is  friendly,  hostile,  or  disinterested). 

Implications.  The  most  important  aspect  of  the  message  as  it  pertains  to  the 
acceptance  of  external  feedback  is  the  sign,  positive  or  negative.  Once  again,  people  are 
more  likely  to  accept  positive  feedback  and  reject  negative  feedback.  It  has  also  been 
found  that  message  consistency  across  time  affects  acceptance,  as  consistently  positive 
(or  negative)  feedback  is  given,  the  opposite  type  of  feedback  will  be  harder  to  accept 
when  offered.  To  increase  the  acceptance  of  the  feedback,  it  should  always  be 
accompanied  by  some  form  of  example  or  critical  incident  exemplifying  the  feedback. 

The  recipient's  personal  characteristics  can  also  alter  the  acceptance  of  external 
feedback,  as  people  with  an  internal  locus  of  control  will  be  more  likely  to  accept  the 
feedback  and  act  upon  it  versus  people  with  external  loci  of  control  (Feather,  1968).  Age 
can  impact  the  acceptance  of  feedback,  as  older  and  more  experienced  individuals  are 
less  likely  to  accept  feedback  than  younger,  less  experienced  individuals  (Meyer  & 
Walker  1961). 

Desire  to  respond.  After  the  acceptance  stage,  participants  often  feel  the  desire  to 
respond.  The  desire  to  respond  to  the  feedback  concerns  the  individual  having  the  choice 
to  act  upon  received  feedback,  either  making  the  behavioral  changes  or  not.  The 
recipient  plays  a  large  role  in  this  aspect.  Expectancy  theory  suggests  that  individuals 
with  high  self-efficacy  will  increase  the  chances  of  action  on  received  feedback 
(Campbell  &  Pritchard,  1976).  Motivation  of  the  individual  can  also  impact  the  desire  to 
respond,  as  feedback  allows  the  recipient  to  judge  his  or  her  perfonnance.  If  the 
performance  is  judged  to  be  proficient,  then  feelings  of  competence  will  increase,  which 
is  a  strong  motivator  (White,  1959).  Personal  level  of  control  can  dictate  one's  desire  to 
respond,  with  individuals  who  attribute  behavior  changes  to  their  own  will  showing  a 
greater  likelihood  of  responding.  Alternatively,  those  who  feel  controlled  by  the  source 
of  the  feedback  will  decrease  in  desire  to  respond  (Deci,  1975).  For  example,  leaders 
who  structure  behavior  will  promote  satisfaction  only  if  the  group  lacks  structure; 
however  if  the  structuring  is  redundant,  it  will  be  received  as  controlling  and  cause 
dissatisfaction  (House,  1971). 
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Implications.  ,  Feedback  should  be  designed  to  bolster  the  recipients’  feelings  of 
competency,  while  reducing  the  feeling  of  external  control  to  be  most  effective.  The 
feedback  should  generally  be  positive,  and  provide  information  above  and  beyond  the 
information  already  retained  by  the  recipient. 

Intended  response.  The  last  stage  of  the  feedback  process  is  the  intended 
response  step.  The  desire  to  respond  to  feedback  does  not  always  mean  an  actual 
response,  as  there  are  internal  and  external  factors  that  may  interrupt  the  response  after 
the  decision  has  been  made  to  make  it.  The  intended  response  ties  in  heavily  with  goal 
setting  and  checking,  as  feedback  is  essential  to  the  goal  process.  Specific  goals  lead  to  a 
more  substantial  increase  in  performance  compared  to  general  goals  which  suffer  from 
limited  functionality  (Steers  &  Porter,  1974).  Difficult  goals  also  increase  perfonnance 
in  comparison  to  easy  ones.  The  more  personal  control  over  goals  set,  the  more 
performance  will  increase.  To  support  specific  goals,  specific  feedback  should  be 
provided,  and  done  so  in  meaningful  chunks  of  information  that  the  recipient  will  be  able 
to  act  upon. 

Implications.  What  this  four-step  process  suggests  for  the  AAR  is  that  the 
facilitator  attempt  to  ensure  that  each  person  in  the  AAR  agrees  with  the  feedback  that  is 
given  so  that  it  will  not  be  disregarded.  Feedback  should  also  be  delivered  in  a  timely 
fashion;  the  general  feedback  messages  should  be  frequent  and  consistent.  Feedback 
generated  by  the  AAR  participants  about  themselves  will  be  accepted  more  readily,  so  the 
facilitator  should  try  to  have  the  participants  generate  their  own  personal  feedback  when 
addressing  issues.  Facilitators  should  try,  when  possible,  to  avoid  delivering  excessive 
negative  feedback,  as  it  is  generally  not  as  effective  as  positive.  Positive  feedback  will 
boost  the  participants’  feelings  of  competency,  which  will  increase  motivation  to 
perform.  AAR  facilitators  should  work  hard  to  maintain  themselves  as  credible, 
energetic,  reliable,  and  representing  the  participants’  best  interests.  The  AAR  designer 
and  AAR  facilitator  should  also  help  the  participants  develop  challenging,  specific  goals 
for  future  perfonnance,  and  subsequently  offer  specific  feedback  to  provide  a  means  of 
monitoring  progress  towards  these  goals. 

Recommendations 

From  the  research  findings  that  were  discussed  by  Newlin  and  Bliss  (2007)  as  part 
of  their  ITAAR  report,  and  from  the  research  findings  discussed  in  this  report,  several 
conclusions  are  worthy  of  discussion.  In  the  following  paragraphs,  we  discuss  the  AAR 
as  a  subject  of  research.  The  initial  section  will  discuss  major  centers  of  AAR  research. 
Subsequent  sections  will  include  lessons  learned  across  research  domains  and  fields, 
areas  that  have  yet  to  be  investigated,  and  potential  avenues  for  further  work  regarding 
AAR. 

Our  literature  review  revealed  that  there  are  a  number  of  research  centers  (military 
and  non-military)  that  have  actively  studied  the  effectiveness  of  AARs.  In  some  cases, 
the  driving  force  behind  such  work  is  the  improvement  of  the  process  for  theoretical 
purposes.  This  could  be  referred  to  as  “basic  research”  with  regard  to  the  AAR.  Given 
the  wide  variety  of  psychological  and  social  constructs  impacting  the  delivery  of  training 
feedback  (some  of  which  have  been  reported  in  this  document),  it  is  not  surprising  that 
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certain  research  centers  have  focused  on  acquisition  and  dissemination  of  knowledge  as 
their  primary  goal. 

Other  centers  have  focused  on  improvement  of  the  AAR  process  from  a  logistic 
standpoint.  More  specifically,  these  centers  are  concerned  mostly  with  how  AARs  might 
be  designed  to  benefit  the  trainees  and  facilitators  who  must  use  them.  Questions 
pertinent  to  this  line  of  inquiry  often  concern  the  optimal  duration,  foci,  environment,  and 
material  to  be  addressed  within  the  AAR.  Often,  aspects  of  leadership  (with  regard  to  the 
AAR  facilitator)  may  arise  here  as  well,  along  with  the  more  basic  investigations  of  the 
AAR  process. 

A  third  category  of  research  center  revolves  around  the  examination  of  AARs  for 
the  purpose  of  generalization.  As  an  increasing  number  and  variety  of  organizations 
adopt  the  AAR  for  their  particular  purposes,  questions  about  duration,  focus,  and 
interaction  style  must  be  addressed.  Unfortunately,  many  organizations  have  not 
considered  such  questions  in  sufficient  detail.  As  a  result,  the  application  of  prior  AAR 
methodology  to  specialized  training  scenarios  has  been  inefficient  at  best  and  at  times 
c  ounterproductive . 

As  a  result  of  the  diaspora  of  the  AAR  concept  that  occurred  in  the  1990s,  allied 
military  institutions  began  to  adopt  the  AAR  for  their  own  use.  Prolific  examples  of 
research  activity  may  be  found  within  literature  published  by  (at  a  minimum)  the  Air 
Force,  Navy,  Marines,  and  Coast  Guard.  The  U.S.  Army  Center  for  Army  Lessons 
Learned  (CALL)  has  as  its  mission  the  improvement  of  perfonnance;  for  that  reason,  they 
are  among  the  most  invested  agencies  in  the  AAR  process. 

Outside  of  the  military,  there  have  been  a  number  of  organizations  in  the 
commercial  sector  that  have  studied  the  AAR  process.  Many  of  these  organizations  view 
the  AAR  as  a  panacea  for  the  low-performing  organization.  In  fact,  the  Harvard  Business 
School  has  included  methodology  and  philosophy  of  AARs  as  essential  reading  for  the 
study  of  high-performing  organizations  (Garvin,  2000).  Many  of  the  organizations  are 
consulting  centers,  catering  to  the  military,  academic,  and  business  world  in  their  quest 
for  high  performing  teams.  A  large  variety  of  academic  institutions  have  also  been  active 
in  the  study  of  AARs.  In  most  cases,  such  institutions  have  been  those  with  ties  to  the 
academic  fields  of  human  factors,  industrial  psychology,  industrial  engineering, 
instructional  design,  or  training.  Frequently,  such  schools  will  also  be  co-located  with 
military  centers  or  units. 

In  recent  years,  there  have  been  a  number  of  agencies  that  have  begun  to  look 
strongly  at  the  AAR  process.  The  military  remains  invested  in  the  process,  due  to 
unconventional  conflicts  being  pursued  in  Afghanistan,  Iraq,  and  Bosnia.  In  addition,  the 
reaction  of  aid  relief  centers  to  natural  disasters  such  as  Hurricanes  Katrina,  Rita,  and  Ike 
have  caused  a  renewed  interest  in  the  AAR  procedure  to  understand  the  fallibilities  of  the 
emergency  response  system.  In  fact,  Pulliam  (2006)  noted  that  President  Bush  had 
ordered  an  investigative  AAR  process  to  be  completed  in  the  wake  of  Hurricane  Katrina. 
In  addition,  as  noted  earlier  in  this  report,  the  Wildland  Fire  Safety  Association  has 
developed  its  own  AAR  process,  based  in  large  part  on  the  U.S.  Army’s  approach.  As 
businesses  and  agencies  continue  to  grapple  with  global  economic  challenges,  it  is 
foreseeable  that  the  AAR  procedure  will  be  adopted  by  even  more  agencies.  Such 
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attention  may  result  in  the  growth  and  maturation  of  the  AAR.  Research  has  already 
been  completed  to  facilitate  the  automation  of  the  AAR  process  (Chen,  Jensen,  Bascara, 
&  Harmon,  2007). 

Lessons  Learned  Across  Research  Domains  or  Fields 

The  report  by  Newlin  and  Bliss  (2007)  included  a  number  of  research-driven 
recommendations  for  the  optimal  conduct  of  AARs.  The  majority  of  those 
recommendations  have  been  reproduced  as  Appendix  A  of  this  report.  Further 
conclusions  may  be  drawn  from  the  research  that  has  been  presented  here.  The  following 
bullet  points  are  meant  to  summarize  the  theoretical  information  presented  earlier. 

•  AAR  designers  and  facilitators  should  arrange  and  implement  AAR  sessions  to 
ensure  the  prompt  encoding,  robust  storage,  and  ready  retrieval  of  task  relevant 
information  by  trainees.  An  example  of  such  a  process  may  involve  exploiting 
human  tendencies  for  spreading  activation  by  pointing  out  relationships  among 
mission  items,  goals  and  procedures. 

•  To  enable  deep  and  lasting  processing  of  mission  information,  AAR  facilitators 
should  arrange  for  presentation  of  “ground  truth”  from  a  variety  of  perspectives. 

•  Facilitators  should  use  hierarchical  review  of  mission  events  to  cater  to  the 
development  of  cognitive  scripts,  mental  models,  and  cognitive  maps. 

•  To  prevent  false  memory  formation,  facilitators  should  encourage  checks  and 
balances  among  team  members,  along  with  presentations  of  recorded  mission 
action. 

•  Facilitators  should  take  care  to  address  the  need  for  shared  situation  awareness 
among  team  members. 

•  Team  members  should  be  encouraged  to  express  the  benefits  gained  by  review  of 
team  perfonnance. 

•  Demands  for  divided  attention  should  be  balanced  with  the  growth  of  expertise  by 
presenting  material  in  a  graduated  (adaptive)  and  distributed  fashion. 

•  Facilitators  should  be  flexible  in  team  assignments  and  feedback,  focusing  on 
dyads  or  smaller  team  units  to  illustrate  important  points. 

•  Facilitators  should  be  prepared  to  rehearse  contingency  leadership  plans  in 
situations  where  attrition  occurs. 

•  Facilitators  and  team  members  should  be  sensitive  to  communication 
requirements  and  breakdowns.  Social  dynamics  such  as  attributions,  conflict,  and 
cooperation  are  central  to  a  team’s  ultimate  success. 
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•  Teams  and  facilitators  must  be  aware  of  mission  tempo,  to  take  advantage  of 
“down  times”  for  reorganization  and  planning. 

•  Feedback  sessions  should  allow  for  self-feedback  and  task  feedback  possibilities. 

•  Facilitators  should  be  aware  of  the  potential  for  negative  reactions  to  feedback, 
and  plan  accordingly. 

•  Where  possible,  feedback  items  should  be  accompanied  by  critical  incidents. 

•  Facilitators  should  ensure  that  all  team  members  accept  the  feedback  given  during 
the  AAR  session.  If  dissent  exists,  work  to  resolve  the  conflict. 

Need  for  Additional  Research 

The  burgeoning  interest  in  AAR  design  and  implementation  has  led  to 
advancement  in  many  areas.  Yet,  given  the  complexity  of  the  AAR  process  and  the 
advancements  being  made  in  simulation  and  training  and  automated  task  delegation,  we 
believe  that  additional  research  is  warranted  in  several  areas.  In  some  of  these  areas, 
research  has  been  conducted  by  a  limited  number  of  scientists,  trainers,  and  institutions. 
In  others,  little  work  has  been  attempted. 

Distributed  AAR  -  As  the  nature  of  warfare  changes  to  accommodate  a  variety  of 
battle  scenarios,  Soldiers  often  find  themselves  fighting  an  unconventional  foe  that  relies 
on  unorthodox  methods  to  achieve  its  goals.  Additionally,  military  forces  are  often 
required  to  win  the  hearts  and  minds  of  citizens  and  governments  by  engaging  in 
peacekeeping,  peace  enforcement,  and  humanitarian  missions.  Such  a  diversity  of  duties 
frequently  requires  that  teams  of  Soldiers  be  formed  from  several  centers.  The  result 
reflects  a  joint  philosophy  of  warfare,  and  requires  novel  training  methods  to  ensure  that 
Soldiers  can  learn  quickly  and  perform  their  duties  effectively.  Modern  simulation 
technology  has  offered  the  capability  to  train  tasks  to  teams  that  are  geographically 
distributed.  Yet  AARs  are  a  piece  of  the  training  puzzle  that  has  yet  to  be  refined  for 
application  to  geographically  dispersed  teams.  Some  systems  are  in  place  to  attempt  this 
(Lampton,  Bliss,  Kring,  Martin,  Saffold,  &  Garrity,  2008);  however,  the  emphasis  has 
been  largely  on  technological  capability.  If  successfully  used,  distributed  AAR  systems 
may  allow  performance  to  be  enhanced.  However,  a  considerable  number  of  social  and 
knowledge  management  issues  must  be  considered  before  seamless  implementation  is 
achieved. 

Shared  mental  models  -  As  we  investigated  the  relevant  literature  with  regard  to 
cognition,  team  training  and  situation  awareness,  we  realized  that  these  three  areas  were 
of  extreme  importance  to  the  success  of  AARs,  and  that  they  intersected  through  one 
major  construct:  shared  mental  models.  In  many  cases,  AAR  facilitators  must  lead  team 
members  to  understand  the  battle  scenario  from  a  common  point  of  view.  They  must  also 
encourage  individual  team  members  to  appreciate  their  teammates’  points  of  view.  For 
both  of  these  goals  to  be  achieved,  it  is  necessary  to  understand  how  team  members  can 
develop  a  shared  mental  model  of  the  battle,  the  terrain,  the  opposing  force,  and  the 
guidelines  that  shape  the  encounter.  Shared  mental  models  have  been  a  popular  topic  for 
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researchers  to  explore  for  many  years  (Banks  &  Millward,  2007;  Cannon-Bowers,  Salas, 
&  Converse,  1993;  Minionis,  Zaccaro,  &  Perez,  1995).  However,  such  investigations 
have  not  always  targeted  the  AAR  process.  Such  a  process  is  unique  because  of  its 
complexity  and  its  focus  on  instructor-led  instruction.  We  believe  it  is  imperative  to 
study  variables  specific  to  the  AAR  experience  to  understand  how  they  might  influence 
the  acquisition  of  shared  mental  models.  Such  variables  may  include  (to  name  a  few) 
inter-team  competition,  conflicting  goals,  attrition  of  team  members,  and  spontaneous 
collaboration  with  outside  leaders  or  teams. 

Another  area  that  is  related  to  shared  mental  models  is  the  notion  of  shared 
situation  awareness.  Though  not  emphasized  strongly  in  this  report,  it  is  clear  that  a 
desired  product  of  the  AAR  session  is  a  common  appreciation  for  the  battle  situation  and 
the  influences  on  it.  Of  particular  importance  are  the  various  levels  of  situation 
awareness  and  how  they  may  change  given  tactics  and  strategies  employed  by  the 
trainees.  Military  teams,  like  many  other  teams,  are  encouraged  to  anticipate  conflicts 
and  obstacles  to  success.  To  do  so,  it  is  imperative  that  members  maintain  shared  level  3 
situation  awareness.  However,  communication  is  essential  for  the  fonnation  of  such  a 
commodity.  Frequently,  members  of  a  fire  team  may  experience  different  aspects  of  a 
conflict.  Similar  to  the  blind  men  describing  the  elephant,  they  may  each  hold  a  piece  of 
the  puzzle,  so  that  only  by  sharing  their  information  might  they  all  enjoy  an  accurate 
prognostication  of  future  events.  Investigating  shared  situation  awareness  may  entail 
examinations  of  technology-driven  environmental  displays  (of  terrain,  forces,  or  supplies) 
and  may  require  consideration  of  past  perfonnance  and  events. 

A  further  area  that  has  enjoyed  considerable  research  is  leadership.  Within  this 
report,  we  have  made  several  recommendations  that  concern  the  leadership  behaviors 
demonstrated  by  AAR  facilitators.  Though  researchers  have  studied  leadership  behaviors 
for  many  years  (dating  back,  perhaps  to  the  Ohio  State  studies  of  leadership;  see 
Fleishman,  1953),  the  AAR  environment  introduces  aspects  of  contingency-based 
leadership  that  may  be  most  relevant  for  modern  theories.  Yet,  certain  modem  theories 
of  leadership  such  as  transformational  leadership  and  leader-member  exchange  theory 
may  not  apply  to  the  AAR  situation,  due  to  the  scripted  nature  of  training  missions  and 
trainee  roles  and  behaviors.  Interestingly  enough,  the  AAR  process  bears  a  similarity  to  a 
number  of  other  leadership  situations  such  as  sports  teams  or  business-related  “tiger 
teams”,  where  problems  are  relatively  well-defined  and  members  have  acknowledged 
areas  of  expertise.  Future  researchers  may  do  well  to  explore  the  practice  of  effective 
leaders  so  that  facilitator  actions  may  be  prescribed  and  perhaps  ultimately  automated. 

A  final  area  that  warrants  study  seems  obvious  enough  to  raise  eyebrows.  Yet, 
though  the  AAR  has  been  an  Army  tradition  for  more  than  30  years,  investigations  of  its 
effectiveness  have  been  few  and  anecdotal.  As  is  reflected  within  our  review  of  the 
literature,  the  theoretical  substrates  of  the  AAR  have  been  investigated  to  a  similar 
conclusion:  a  structured  feedback  session  that  includes  collective  review  of  performance 
and  guided  improvement  steps  should  be  successful  for  perfonnance  refinement. 
However,  what  are  missing  are  rigorous,  empirical  demonstrations  of  this  conclusion.  It 
is  necessary,  now  more  than  ever  before,  that  experimenters  demonstrate  that  well- 
designed  and  meticulously  implemented  AARs  can  lead  to  improvements  in  training. 

Such  investigations  must  go  beyond  collecting  subjective  opinions  and  general  statements 
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of  approval  voiced  by  command  elements.  Rather,  researchers  must  complete  a  program 
of  transfer  of  training  studies  to  show  that  variations  in  AAR  design  and  conduct  will 
cause  differences  in  individual  and  collective  task  performance.  Naturally,  studies  such 
as  these  have  been  extremely  difficult  to  complete  because  of  the  variability  associated 
with  human  facilitators,  human  trainees,  complex  missions,  and  logistic  constraints. 
However,  with  the  growth  of  virtual  battlespaces  and  tools  for  reviewing  and 
documenting  training  perfonnances,  the  ability  now  exists  to  accomplish  this  goal.  An 
associated  but  loftier  goal  would  be  the  documentation  of  relative  benefit  linked  to 
changes  in  facilitator  technique  and  AAR  content.  Showing  such  a  linkage  would  allow 
commercial,  governmental  and  academic  enterprises  to  adopt  the  AAR  method  and  reap 
its  benefits  more  reliably. 

In  conclusion,  much  work  has  been  completed  to  support  the  design  and 
implementation  of  AARs  for  military  and  non-military  purposes.  However,  those  who 
have  employed  AARs  have  not  been  content  to  wait  for  analysts  to  complete  their  work. 
Instead,  event  trainers  and  investigators  have  proceeded  with  AAR  implementation,  using 
the  technique  to  justify  policy  shifts,  design  decisions,  and  implementation  of  strategies 
for  training.  It  is  now  time  to  integrate  the  underlying  theories  into  a  general  theory  of 
AAR.  Doing  so  may  allow  researchers  to  better  study  the  technique,  and  will  hopefully 
encourage  AAR  designers  to  consider  the  implications  of  their  decisions.  Our  research 
team  faces  a  clear  path  forward:  we  plan  to  validate  the  IT  AAR  model,  noting  where  it 
converges  and  diverges  with  real-world  AAR  practice  and  established  theory.  We  will 
then  be  poised  to  make  clear  recommendations  for  its  improvement  and  use. 
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Appendix  A:  Recommendations  (from  Newlin  &  Bliss,  2007) 


Recommendations  for  the  Facilitator’s  Role  in  AAR 


Topic 

Source 

Recommendation 

Facilitator’s 

Role 

Dyer,  1986 

Facilitators  should  not  interrupt  during  training  exercises. 

Evaluation  aids  help  the  instructor  monitor  actions  that  are 
indicative  of  good  trainee  coordination  and  communication. 

Diagnostic  tools  assess  trainee  knowledge  prior  to  training 
assessment  and 

Using  a  standardized  assessment  of  team  performance 
periodically  during  the  training  exercise. 

Atwater,  Rousch,  & 
Fischthal,  1995 

A  360  degree  evaluation  helps  the  facilitator  maintain  objectivity 
and  improve  instructional  methods. 

Borman,  1997 

A  360  degree  evaluation  provides  the  facilitator  with  information 
about  their  abilities  as  well  as  group  training  performance. 

Effective 

Leadership 

Borich,  1989 

The  facilitator  should  clearly  communicate  with  trainees. 

The  facilitator  should  be  capable  of  adjusting  their 
communication  style  to  fit  the  trainees’  level  of  knowledge. 

Facilitators  should  be  task  oriented. 

Effective  facilitators  should  incorporate  all  trainees  in  training. 

Schmidt  &  Kleinbeck, 

1990 

Facilitators  should  formally  state  their  expectations  in  an 
instructional  plan  so  the  trainees  understand  their  learning  goals. 

Mager,  1984 

Learning  goals  should  describe  the  appropriate  action,  clearly 
identify  the  environment  and  accessible  tools,  and  the  acceptable 
parameters  for  performance. 

Facilitators  should  provide  feedback  that  compares  training 
perfonnance  to  goal  criteria. 

Rosenthal  &  Jacobson, 
1968;  Eden  &  Shani, 

1982;  Hoyt,  Halverson, 
Murphy  &  Watson,  2003 

Facilitators  should  be  aware  that  their  expectations  improved 
perfonnance. 

Gobet,  1998 

Facilitators  should  possess  expert  knowledge  because  they  must 
quickly  identify  critical  events  during  training. 

A-l 


The 

facilitator’s 
role  in 
feedback 


Spencer  &  Spencer,  1993; 
Raynor  &  Rubin,  1971 

The  facilitator  should  not  be  the  only  feedback  source. 

Trainees  are  more  engaged  in  learning,  post  training  discussions, 
and  generating  feedback  when  the  facilitator  addresses  their 
needs. 

An  AAR  facilitator  should  communicate  to  group  members  that 
their  opinions  are  essential  to  the  AAR  process. 

Watkins,  1990 

The  AAR  facilitator  should  utilize  trainees  when  generating 
feedback  and  guiding  discussions. 

Facilitators  should  perceive  training  as  an  opportunity  to  learn 
from  trainees. 

Marquardt,  1999 

In  action  learning,  the  facilitator  should  teach  trainees  to 
evaluate  their  individual  and  group  performances  and  generate 
effective  feedback. 

The  facilitator  should  be  aware  of  how  trainees’  assumptions 
influence  their  ability  to  identify  problems  and  develop 
solutions. 

Dyer,  1986 

A  facilitator  should  effectively  communicate  information  to 
learners  and  appear  open  to  input  from  trainees. 

Smith,  2001 

Those  with  implicit  knowledge  should  concretely  describe  his  or 
her  implicit  knowledge  when  information  is  changed  from 
implicit  to  explicit  form.  This  is  done  by  experiencing  one’s 
state  of  mind  while  perfonning  a  task  and  then  explaining  their 
thought  processes. 

When  converting  explicit  knowledge  to  implicit  knowledge,  the 
knowledge  should  be  specifically  framed  to  fit  the  trainee’s 
frame  of  reference. 

Bandura,  1977;  Hoyt  et 
ah, 2003 

A  facilitator  should  have  confidence  in  their  abilities  because 
this  affects  how  they  lead  and  how  a  group  of  trainees  perfonn. 

Bandura,  1977;  Goldstein 
and  Sorcher,  1974 

Facilitators  should  model  behaviors  that  generally  promote 
better  performance. 

Gist,  1989 

Facilitators  should  encourage  trainees  to  use  analytical  thinking 
to  brainstonn  problems. 

The  facilitator  should  encourage  trainees  to  entertain  all  ideas 
during  brainstorming,  despite  their  acceptability. 

Trainees  should  express  their  self-generated  solutions  with  other 
group  members  through  writing  or  verbalizing. 

Facilitators  should  encourage  trainees  to  counteract  a  negative 
thought  about  their  ideas  with  a  positive  one. 

A-2 


Recommendations  for  Groups  of  Trainees  in  AAR 


Topic 

Source 

Recommendation 

Effective  teams 

Gladstein,  1984 

Arrange  trainees  to  optimize  positive  group  member 
interactions. 

Sundstrom,  DeMeuse  & 

Futrell,  1990 

Provide  trainees  with  a  clear  mission  statement.  This 
should  refer  to  their  purpose  within  the  larger 
organization. 

Borman,  1997 

A  360  degree  evaluation  provides  the  instructor  with 
infonnation  about  their  abilities  as  well  as  group  training 
perfonnance. 

Factors  that  affect 
group 

performance 

Matsui,  Kakuyama  & 

Onglatco,  1987 

Collective  group  feedback  improves  group  training 
perfonnance. 

Individual  and  group  feedback  also  prevents  social 
loafing  and  improves  both  individual  and  group 
perfonnance. 

Matsui  et  al.,  1987;  Saavedra, 
Earley  &  Van  Dyne,  1993; 

Eocke  &  Latham,  2002 

Goals  should  relate  to  the  training  task  and  the  feedback 
should  describe  how  training  performance  meets  training 
goals. 

Group  goals  plus  individual  goals  are  conducive  to 
group  perfonnance  when  the  feedback  reflects  the  goals. 

Setting  group  goals  causes  trainees  to  collectively 
generate  solutions  and  strategize  about  how  they  can 
improve  performance. 

Specific  and  challenging  goals  often  generate  better 
perfonnance. 

Kiggundu,  1981 

Trainees  should  organize  their  group  to  maximize  team 
members’  abilities. 

Lim  &  Klein,  2006 

Shared  mental  models  must  contain  accurate  information 
about  the  procedures,  technology  and  environment. 

All  group  members  should  share  similar  information. 

A-3 


Topic 

Source 

Recommendation 

Cannon-Bowers  &  Salas,  2001 

Group  members  should  share  similar  performance 
expectations  because  there  are  certain  protocols  and 
procedures  associated  with  the  task. 

Team  members  should  have  the  same  task  performance 
knowledge. 

Team  members  should  know  their  teammates’ 
weaknesses  and  proficiencies  and  how  these  are 
distributed  among  the  group. 

Team  members  are  generally  more  effective  when  they 
have  similar  perceptions  of  the  same  events.  This 
includes  sharing  similar  attitudes  and  beliefs  of  a  variety 
of  tasks. 

Saavedra,  Earley  &  Van  Dyne, 
1993 

Group  members  who  share  infonnation,  collectively 
discuss  problems  and  develop  solutions  will  be  more 
cohesive. 

Learning 

Fitts,  1962;  Gobet,  1998, 

Logan,  1988;  Gonzalez,  Lerch 
&  Lebiere,  2003;  Patrick,  1992 

Trainees  should  verbalize  newly  acquired  knowledge. 

Trainees  should  fixate  on  newly  acquired  knowledge 
until  all  errors  are  eliminated. 

Increased  practice  leads  to  increased  automaticity  and 
generalization. 

Once  skills  become  automatic,  trainees  should  focus  on 
increasing  speed  and  efficiency  despite  multiple  tasks, 
task  complexity  or  time  pressure. 

Hendrickson  &  Schroeder, 

1941 

Trainees  should  be  given  theoretical  information  about 
the  training  task. 

Theoretical  explanations  during  training  must  be 
applicable  and  relate  to  a  task. 

In  AAR,  theoretical  explanations  should  explain  why 
training  tasks  are  important  in  the  overall  scheme  of  the 
learning  task. 

Marquardt,  1999 

Trainees  should  collectively  create  solutions  to  problems 
while  learning  task  skills. 

Each  team  member  should  actively  contribute  to  the 
group’s  learning  by  problem  solving  and  discussing 
group  performance. 

Team  members  should  collectively  interpret 
perfonnance  data,  thereby  creating  consensus  and  group 
cohesion. 

A-4 


Recommendations  for  Performance  Appraisal 


Topic 

Source 

Recommendation 

Purpose  of  the 
Performance  Appraisal 

Patrick,  1992 

The  perfonnance  appraisal  should  identify  which  training 
elements  benefit  training  performance,  so  adjustments 
can  be  made  to  the  training  process. 

By  manipulating  training  variables,  training  elements  are 
isolated  and  assessed. 

Objective  and 

Qualitative  Measures 
used  in  Performance 
Appraisal 

Kahn,  1977  &  Jones, 

1997 

Objective  measures  can  evaluate  performance  and 
compare  performance  to  team  goals. 

Kinlaw,  1991;  Coulter, 
1979;  Scott,  1977 

To  evaluate  group  functioning,  surveys  should  include 
questions  about  team  members’  feelings  of  cooperation, 
group  cohesion,  and  flexibility. 

These  questions  should  assess  group  members’  open 
mindedness  in  discussions  and  responsiveness  to 
criticism. 

Borman  and  Motowidlo, 
1993;  Levy  &  Steelman, 
1997;  Cohen,  1994 

Less  traditional  measures  provide  a  more  complete 
assessment  of  AAR  effectiveness  when  coupled  with 
objective  measures. 

AAR  performance  appraisals  should  evaluate  team 
members’  interactions  and  cooperation  with  others,  their 
willingness  to  participate  in  discussions,  and  the 
facilitators’  eagerness  to  teach. 

Cohen,  1994 

Appraisals  should  include  other  elements  that  affect 
group  effectiveness:  task  inherent  characteristics,  team 
member  organization,  self  management  and  group 
member  encouragement. 

Nieva,  Fleishman  and 
Rieck  (1978) 

Appraise  team  design  and  how  infonnation  is  dispersed 
among  team  members.  This  includes  how  trainees  share 
resources  and  knowledge. 

Assess  how  group  members  assigned  and  completed 
tasks.  Groups  should  assign  tasks  based  on  their 
knowledge  of  group  member’s  abilities. 

Evaluate  how  group  members  collectively  identify 
performance  errors  and  problems. 

Hackman,  1987 

Evaluate  the  effect  of  team  member  interactions  on  other 
group  members. 

These  interactions  should  encourage  and  fulfill 
individual  group  members. 
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Appraisal 

Levy  and  Steelman,  1997; 

Appraisal  methods  should  combine  elements  of  individual  and 

Systems 

Matsui  et  al.,  1987 

group  assessment. 

Using  multiple  raters,  additional  peer  and  self  evaluations 

eliminates  the  problems  associated  with  a  biased  evaluator. 

A-6 


